Merge branch 'master' of github.com:MongoEngine/mongoengine into add_validation_to_doc

docs/guide/connecting.rst

@@ -86,7 +86,7 @@ using 3 different databases to store data::
         connect(alias='user-db-alias', db='user-db')
         connect(alias='book-db-alias', db='book-db')
         connect(alias='users-books-db-alias', db='users-books-db')
-        
+
         class User(Document):
             name = StringField()
 

docs/guide/defining-documents.rst

@@ -352,7 +352,7 @@ Its value can take any of the following constants:
   Deletion is denied if there still exist references to the object being
   deleted.
 :const:`mongoengine.NULLIFY`
-  Any object's fields still referring to the object being deleted are removed
+  Any object's fields still referring to the object being deleted are set to None
   (using MongoDB's "unset" operation), effectively nullifying the relationship.
 :const:`mongoengine.CASCADE`
   Any object containing fields that are referring to the object being deleted
@@ -555,7 +555,6 @@ There are a few top level defaults for all indexes that can be set::
             'index_background': True,
             'index_cls': False,
             'auto_create_index': True,
-            'index_drop_dups': True,
         }
 
 
@@ -574,11 +573,6 @@ There are a few top level defaults for all indexes that can be set::
     in systems where indexes are managed separately. Disabling this will improve
     performance.
 
-:attr:`index_drop_dups` (Optional)
-    Set the default value for if an index should drop duplicates
-    Since MongoDB 3.0 drop_dups is not supported anymore. Raises a Warning
-    and has no effect
-
 
 Compound Indexes and Indexing sub documents
 -------------------------------------------
@@ -714,11 +708,16 @@ subsequent calls to :meth:`~mongoengine.queryset.QuerySet.order_by`. ::
 Shard keys
 ==========
 
-If your collection is sharded, then you need to specify the shard key as a tuple,
-using the :attr:`shard_key` attribute of :attr:`~mongoengine.Document.meta`.
-This ensures that the shard key is sent with the query when calling the
-:meth:`~mongoengine.document.Document.save` or
-:meth:`~mongoengine.document.Document.update` method on an existing
+If your collection is sharded by multiple keys, then you can improve shard
+routing (and thus the performance of your application) by specifying the shard
+key, using the :attr:`shard_key` attribute of
+:attr:`~mongoengine.Document.meta`. The shard key should be defined as a tuple.
+
+This ensures that the full shard key is sent with the query when calling
+methods such as :meth:`~mongoengine.document.Document.save`,
+:meth:`~mongoengine.document.Document.update`,
+:meth:`~mongoengine.document.Document.modify`, or
+:meth:`~mongoengine.document.Document.delete` on an existing
 :class:`~mongoengine.Document` instance::
 
     class LogEntry(Document):
@@ -728,7 +727,8 @@ This ensures that the shard key is sent with the query when calling the
         data = StringField()
 
         meta = {
-            'shard_key': ('machine', 'timestamp',)
+            'shard_key': ('machine', 'timestamp'),
+            'indexes': ('machine', 'timestamp'),
         }
 
 .. _document-inheritance:
@@ -738,7 +738,7 @@ Document inheritance
 
 To create a specialised type of a :class:`~mongoengine.Document` you have
 defined, you may subclass it and add any extra fields or methods you may need.
-As this is new class is not a direct subclass of
+As this new class is not a direct subclass of
 :class:`~mongoengine.Document`, it will not be stored in its own collection; it
 will use the same collection as its superclass uses. This allows for more
 convenient and efficient retrieval of related documents -- all you need do is
@@ -761,6 +761,27 @@ document.::
           Setting :attr:`allow_inheritance` to True should also be used in
           :class:`~mongoengine.EmbeddedDocument` class in case you need to subclass it
 
+When it comes to querying using :attr:`.objects()`, querying `Page.objects()` will query
+both `Page` and `DatedPage` whereas querying `DatedPage` will only query the `DatedPage` documents.
+Behind the scenes, MongoEngine deals with inheritance by adding a :attr:`_cls` attribute that contains
+the class name in every documents. When a document is loaded, MongoEngine checks
+it's :attr:`_cls` attribute and use that class to construct the instance.::
+
+    Page(title='a funky title').save()
+    DatedPage(title='another title', date=datetime.utcnow()).save()
+
+    print(Page.objects().count())         # 2
+    print(DatedPage.objects().count())    # 1
+
+    # print documents in their native form
+    # we remove 'id' to avoid polluting the output with unnecessary detail
+    qs = Page.objects.exclude('id').as_pymongo()
+    print(list(qs))
+    # [
+    #   {'_cls': u 'Page', 'title': 'a funky title'},
+    #   {'_cls': u 'Page.DatedPage', 'title': u 'another title', 'date': datetime.datetime(2019, 12, 13, 20, 16, 59, 993000)}
+    # ]
+
 Working with existing data
 --------------------------
 As MongoEngine no longer defaults to needing :attr:`_cls`, you can quickly and

docs/guide/gridfs.rst

@@ -10,8 +10,9 @@ Writing
 GridFS support comes in the form of the :class:`~mongoengine.fields.FileField` field
 object. This field acts as a file-like object and provides a couple of
 different ways of inserting and retrieving data. Arbitrary metadata such as
-content type can also be stored alongside the files. In the following example,
-a document is created to store details about animals, including a photo::
+content type can also be stored alongside the files. The object returned when accessing a
+FileField is a proxy to `Pymongo's GridFS <https://api.mongodb.com/python/current/examples/gridfs.html#gridfs-example>`_
+In the following example, a document is created to store details about animals, including a photo::
 
     class Animal(Document):
         genus = StringField()
@@ -20,8 +21,8 @@ a document is created to store details about animals, including a photo::
 
     marmot = Animal(genus='Marmota', family='Sciuridae')
 
-    marmot_photo = open('marmot.jpg', 'rb')
-    marmot.photo.put(marmot_photo, content_type = 'image/jpeg')
+    with open('marmot.jpg', 'rb') as fd:
+        marmot.photo.put(fd, content_type = 'image/jpeg')
     marmot.save()
 
 Retrieval
@@ -34,6 +35,20 @@ field. The file can also be retrieved just as easily::
     photo = marmot.photo.read()
     content_type = marmot.photo.content_type
 
+.. note:: If you need to read() the content of a file multiple times, you'll need to "rewind"
+    the file-like object using `seek`::
+
+        marmot = Animal.objects(genus='Marmota').first()
+        content1 = marmot.photo.read()
+        assert content1 != ""
+
+        content2 = marmot.photo.read()    # will be empty
+        assert content2 == ""
+
+        marmot.photo.seek(0)              # rewind the file by setting the current position of the cursor in the file to 0
+        content3 = marmot.photo.read()
+        assert content3 == content1
+
 Streaming
 ---------
 

docs/guide/index.rst

@@ -14,4 +14,5 @@ User Guide
    gridfs
    signals
    text-indexes
+   logging-monitoring
    mongomock

docs/guide/installing.rst

@@ -12,7 +12,7 @@ MongoEngine is available on PyPI, so you can use :program:`pip`:
 
 .. code-block:: console
 
-    $ pip install mongoengine
+    $ python -m pip install mongoengine
 
 Alternatively, if you don't have setuptools installed, `download it from PyPi
 <http://pypi.python.org/pypi/mongoengine/>`_ and run

docs/guide/logging-monitoring.rst

@@ -0,0 +1,80 @@
+==================
+Logging/Monitoring
+==================
+
+It is possible to use `pymongo.monitoring <https://api.mongodb.com/python/current/api/pymongo/monitoring.html>`_ to monitor
+the driver events (e.g: queries, connections, etc). This can be handy if you want to monitor the queries issued by
+MongoEngine to the driver.
+
+To use `pymongo.monitoring` with MongoEngine, you need to make sure that you are registering the listeners
+**before** establishing the database connection (i.e calling `connect`):
+
+The following snippet provides a basic logging of all command events:
+
+.. code-block:: python
+
+    import logging
+    from pymongo import monitoring
+    from mongoengine import *
+
+    log = logging.getLogger()
+    log.setLevel(logging.DEBUG)
+    logging.basicConfig(level=logging.DEBUG)
+
+
+    class CommandLogger(monitoring.CommandListener):
+
+        def started(self, event):
+            log.debug("Command {0.command_name} with request id "
+                     "{0.request_id} started on server "
+                     "{0.connection_id}".format(event))
+
+        def succeeded(self, event):
+            log.debug("Command {0.command_name} with request id "
+                     "{0.request_id} on server {0.connection_id} "
+                     "succeeded in {0.duration_micros} "
+                     "microseconds".format(event))
+
+        def failed(self, event):
+            log.debug("Command {0.command_name} with request id "
+                     "{0.request_id} on server {0.connection_id} "
+                     "failed in {0.duration_micros} "
+                     "microseconds".format(event))
+
+    monitoring.register(CommandLogger())
+
+
+    class Jedi(Document):
+        name = StringField()
+
+
+    connect()
+
+
+    log.info('GO!')
+
+    log.info('Saving an item through MongoEngine...')
+    Jedi(name='Obi-Wan Kenobii').save()
+
+    log.info('Querying through MongoEngine...')
+    obiwan = Jedi.objects.first()
+
+    log.info('Updating through MongoEngine...')
+    obiwan.name = 'Obi-Wan Kenobi'
+    obiwan.save()
+
+
+Executing this prints the following output::
+
+    INFO:root:GO!
+    INFO:root:Saving an item through MongoEngine...
+    DEBUG:root:Command insert with request id 1681692777 started on server ('localhost', 27017)
+    DEBUG:root:Command insert with request id 1681692777 on server ('localhost', 27017) succeeded in 562 microseconds
+    INFO:root:Querying through MongoEngine...
+    DEBUG:root:Command find with request id 1714636915 started on server ('localhost', 27017)
+    DEBUG:root:Command find with request id 1714636915 on server ('localhost', 27017) succeeded in 341 microseconds
+    INFO:root:Updating through MongoEngine...
+    DEBUG:root:Command update with request id 1957747793 started on server ('localhost', 27017)
+    DEBUG:root:Command update with request id 1957747793 on server ('localhost', 27017) succeeded in 455 microseconds
+
+More details can of course be obtained by checking the `event` argument from the `CommandListener`.

docs/guide/mongomock.rst

@@ -2,10 +2,10 @@
 Use mongomock for testing
 ==============================
 
-`mongomock <https://github.com/vmalloc/mongomock/>`_ is a package to do just 
+`mongomock <https://github.com/vmalloc/mongomock/>`_ is a package to do just
 what the name implies, mocking a mongo database.
 
-To use with mongoengine, simply specify mongomock when connecting with 
+To use with mongoengine, simply specify mongomock when connecting with
 mongoengine:
 
 .. code-block:: python
@@ -21,7 +21,7 @@ or with an alias:
     conn = get_connection('testdb')
 
 Example of test file:
---------
+---------------------
 .. code-block:: python
 
     import unittest
@@ -45,4 +45,4 @@ Example of test file:
             pers.save()
 
             fresh_pers = Person.objects().first()
-            self.assertEqual(fresh_pers.name, 'John')
+            assert fresh_pers.name ==  'John'

docs/guide/querying.rst

@@ -222,6 +222,18 @@ keyword argument::
 
 .. versionadded:: 0.4
 
+Sorting/Ordering results
+========================
+It is possible to order the results by 1 or more keys using :meth:`~mongoengine.queryset.QuerySet.order_by`.
+The order may be specified by prepending each of the keys by "+" or "-". Ascending order is assumed if there's no prefix.::
+
+    # Order by ascending date
+    blogs = BlogPost.objects().order_by('date')    # equivalent to .order_by('+date')
+
+    # Order by ascending date first, then descending title
+    blogs = BlogPost.objects().order_by('+date', '-title')
+
+
 Limiting and skipping results
 =============================
 Just as with traditional ORMs, you may limit the number of results returned or
@@ -349,9 +361,9 @@ Just as with limiting and skipping results, there is a method on a
 You could technically use ``len(User.objects)`` to get the same result, but it
 would be significantly slower than :meth:`~mongoengine.queryset.QuerySet.count`.
 When you execute a server-side count query, you let MongoDB do the heavy
-lifting and you receive a single integer over the wire. Meanwhile, len()
+lifting and you receive a single integer over the wire. Meanwhile, ``len()``
 retrieves all the results, places them in a local cache, and finally counts
-them. If we compare the performance of the two operations, len() is much slower
+them. If we compare the performance of the two operations, ``len()`` is much slower
 than :meth:`~mongoengine.queryset.QuerySet.count`.
 
 Further aggregation
@@ -386,6 +398,25 @@ would be generating "tag-clouds"::
     top_tags = sorted(tag_freqs.items(), key=itemgetter(1), reverse=True)[:10]
 
 
+MongoDB aggregation API
+-----------------------
+If you need to run aggregation pipelines, MongoEngine provides an entry point to `Pymongo's aggregation framework <https://api.mongodb.com/python/current/examples/aggregation.html#aggregation-framework>`_
+through :meth:`~mongoengine.queryset.QuerySet.aggregate`. Check out Pymongo's documentation for the syntax and pipeline.
+An example of its use would be::
+
+        class Person(Document):
+            name = StringField()
+
+        Person(name='John').save()
+        Person(name='Bob').save()
+
+        pipeline = [
+            {"$sort" : {"name" : -1}},
+            {"$project": {"_id": 0, "name": {"$toUpper": "$name"}}}
+            ]
+        data = Person.objects().aggregate(pipeline)
+        assert data == [{'name': 'BOB'}, {'name': 'JOHN'}]
+
 Query efficiency and performance
 ================================
 
@@ -578,7 +609,7 @@ to push values with index::
 .. note::
     Currently only top level lists are handled, future versions of mongodb /
     pymongo plan to support nested positional operators.  See `The $ positional
-    operator <http://www.mongodb.org/display/DOCS/Updating#Updating-The%24positionaloperator>`_.
+    operator <https://docs.mongodb.com/manual/tutorial/update-documents/#Updating-The%24positionaloperator>`_.
 
 Server-side javascript execution
 ================================

docs/guide/signals.rst

@@ -44,8 +44,8 @@ Available signals include:
 
 `post_save`
   Called within :meth:`~mongoengine.Document.save` after most actions
-  (validation, insert/update, and cascades, but not clearing dirty flags) have 
-  completed successfully.  Passed the additional boolean keyword argument 
+  (validation, insert/update, and cascades, but not clearing dirty flags) have
+  completed successfully.  Passed the additional boolean keyword argument
   `created` to indicate if the save was an insert or an update.
 
 `pre_delete`

docs/guide/text-indexes.rst

@@ -8,7 +8,7 @@ After MongoDB 2.4 version, supports search documents by text indexes.
 Defining a Document with text index
 ===================================
 Use the *$* prefix to set a text index, Look the declaration::
-  
+
   class News(Document):
       title = StringField()
       content = StringField()
@@ -35,10 +35,10 @@ Saving a document::
        content="Various improvements").save()
 
 Next, start a text search using :attr:`QuerySet.search_text` method::
-  
+
   document = News.objects.search_text('testing').first()
   document.title # may be: "Using mongodb text search"
-  
+
   document = News.objects.search_text('released').first()
   document.title # may be: "MongoEngine 0.9 released"