Merge pull request #2441 from bagerard/remove_useless_cls_var

minor improvement in code

.landscape.yml

@@ -1,17 +0,0 @@
-pylint:
-    disable:
-        # We use this a lot (e.g. via document._meta)
-        - protected-access
-
-    options:
-        additional-builtins:
-            # add long as valid built-ins.
-            - long
-
-pyflakes:
-    disable:
-        # undefined variables are already covered by pylint (and exclude long)
-        - F821
-
-ignore-paths:
-    - benchmark.py

README.rst

@@ -12,10 +12,6 @@ MongoEngine
 .. image:: https://coveralls.io/repos/github/MongoEngine/mongoengine/badge.svg?branch=master
   :target: https://coveralls.io/github/MongoEngine/mongoengine?branch=master
 
-.. image:: https://landscape.io/github/MongoEngine/mongoengine/master/landscape.svg?style=flat
-  :target: https://landscape.io/github/MongoEngine/mongoengine/master
-  :alt: Code Health
-
 .. image:: https://img.shields.io/badge/code%20style-black-000000.svg
   :target: https://github.com/ambv/black
 

docs/Makefile

@@ -33,7 +33,7 @@ clean:
 html:
 	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
 	@echo
-	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
+	@echo "Build finished. Check $(BUILDDIR)/html/index.html"
 
 dirhtml:
 	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml

docs/changelog.rst

@@ -6,17 +6,24 @@ Changelog
 Development
 ===========
 - (Fill this out as you fix issues and develop your features).
+- Fix LazyReferenceField dereferencing in embedded documents #2426
+
+Changes in 0.21.0
+=================
 - Bug fix in DynamicDocument which is not parsing known fields in constructor like Document do #2412
 - When using pymongo >= 3.7, make use of Collection.count_documents instead of Collection.count
     and Cursor.count that got deprecated in pymongo >= 3.7.
     This should have a negative impact on performance of count see Issue #2219
 - Fix a bug that made the queryset drop the read_preference after clone().
 - Remove Py3.5 from CI as it reached EOL and add Python 3.9
-- Fix the behavior of Doc.objects.limit(0) which should return all documents (similar to mongodb) #2311
+- Fix some issues related with db_field/field conflict in constructor #2414
+- BREAKING CHANGE: Fix the behavior of Doc.objects.limit(0) which should return all documents (similar to mongodb) #2311
 - Bug fix in ListField when updating the first item, it was saving the whole list, instead of
-    just replacing the first item (as it's usually done) #2392
+    just replacing the first item (as usually done when updating 1 item of the list) #2392
 - Add EnumField: ``mongoengine.fields.EnumField``
 - Refactoring - Remove useless code related to Document.__only_fields and Queryset.only_fields
+- Fix query transformation regarding special operators #2365
+- Bug Fix: Document.save() fails when shard_key is not _id #2154
 
 Changes in 0.20.0
 =================

docs/guide/gridfs.rst

@@ -2,8 +2,6 @@
 GridFS
 ======
 
-.. versionadded:: 0.4
-
 Writing
 -------
 

docs/guide/index.rst

@@ -14,5 +14,6 @@ User Guide
    gridfs
    signals
    text-indexes
+   migration
    logging-monitoring
    mongomock

docs/guide/migration.rst

@@ -0,0 +1,267 @@
+===================
+Documents migration
+===================
+
+The structure of your documents and their associated mongoengine schemas are likely
+to change over the lifetime of an application. This section provides guidance and
+recommendations on how to deal with migrations.
+
+Due to the very flexible nature of mongodb, migrations of models aren't trivial and
+for people that know about `alembic` for `sqlalchemy`, there is unfortunately no equivalent
+library that will manage the migration in an automatic fashion for mongoengine.
+
+Example 1: Addition of a field
+==============================
+
+Let's start by taking a simple example of a model change and review the different option you
+have to deal with the migration.
+
+Let's assume we start with the following schema and save an instance:
+
+.. code-block:: python
+
+    class User(Document):
+        name = StringField()
+
+    User(name="John Doe").save()
+
+    # print the objects as they exist in mongodb
+    print(User.objects().as_pymongo())    # [{u'_id': ObjectId('5d06b9c3d7c1f18db3e7c874'), u'name': u'John Doe'}]
+
+On the next version of your application, let's now assume that a new field `enabled` gets added to the
+existing ``User`` model with a `default=True`. Thus you simply update the ``User`` class to the following:
+
+.. code-block:: python
+
+    class User(Document):
+        name = StringField(required=True)
+        enabled = BooleaField(default=True)
+
+Without applying any migration, we now reload an object from the database into the ``User`` class
+and checks its `enabled` attribute:
+
+.. code-block:: python
+
+    assert User.objects.count() == 1
+    user = User.objects().first()
+    assert user.enabled is True
+    assert User.objects(enabled=True).count() == 0    # uh?
+    assert User.objects(enabled=False).count() == 0   # uh?
+
+    # this is consistent with what we have in the database
+    # in fact, 'enabled' does not exist
+    print(User.objects().as_pymongo().first())    # {u'_id': ObjectId('5d06b9c3d7c1f18db3e7c874'), u'name': u'John'}
+    assert User.objects(enabled=None).count() == 1
+
+As you can see, even if the document wasn't updated, mongoengine applies the default value seamlessly when it
+loads the pymongo dict into a ``User`` instance. At first sight it looks like you don't need to migrate the
+existing documents when adding new fields but this actually leads to inconsistencies when it comes to querying.
+
+In fact, when querying, mongoengine isn't trying to account for the default value of the new field and so
+if you don't actually migrate the existing documents, you are taking a risk that querying/updating
+will be missing relevant record.
+
+When adding fields/modifying default values, you can use any of the following to do the migration
+as a standalone script:
+
+.. code-block:: python
+
+    # Use mongoengine to set a default value for a given field
+    User.objects().update(enabled=True)
+    # or use pymongo
+    user_coll = User._get_collection()
+    user_coll.update_many({}, {'$set': {'enabled': True}})
+
+
+Example 2: Inheritance change
+=============================
+
+Let's consider the following example:
+
+.. code-block:: python
+
+    class Human(Document):
+        name = StringField()
+        meta = {"allow_inheritance": True}
+
+    class Jedi(Human):
+        dark_side = BooleanField()
+        light_saber_color = StringField()
+
+    Jedi(name="Darth Vader", dark_side=True, light_saber_color="red").save()
+    Jedi(name="Obi Wan Kenobi", dark_side=False, light_saber_color="blue").save()
+
+    assert Human.objects.count() == 2
+    assert Jedi.objects.count() == 2
+
+    # Let's check how these documents got stored in mongodb
+    print(Jedi.objects.as_pymongo())
+    # [
+    #   {'_id': ObjectId('5fac4aaaf61d7fb06046e0f9'), '_cls': 'Human.Jedi', 'name': 'Darth Vader', 'dark_side': True, 'light_saber_color': 'red'},
+    #   {'_id': ObjectId('5fac4ac4f61d7fb06046e0fa'), '_cls': 'Human.Jedi', 'name': 'Obi Wan Kenobi', 'dark_side': False, 'light_saber_color': 'blue'}
+    # ]
+
+As you can observe, when you use inheritance, MongoEngine stores a field named '_cls' behind the scene to keep
+track of the Document class.
+
+Let's now take the scenario that you want to refactor the inheritance schema and:
+- Have the Jedi's with dark_side=True/False become GoodJedi's/DarkSith
+- get rid of the 'dark_side' field
+
+move to the following schemas:
+
+.. code-block:: python
+
+    # unchanged
+    class Human(Document):
+        name = StringField()
+        meta = {"allow_inheritance": True}
+
+    # attribute 'dark_side' removed
+    class GoodJedi(Human):
+        light_saber_color = StringField()
+
+    # new class
+    class BadSith(Human):
+        light_saber_color = StringField()
+
+MongoEngine doesn't know about the change or how to map them with the existing data
+so if you don't apply any migration, you will observe a strange behavior, as if the collection was suddenly
+empty.
+
+.. code-block:: python
+
+    # As a reminder, the documents that we inserted
+    # have the _cls field = 'Human.Jedi'
+
+    # Following has no match
+    # because the query that is used behind the scene is
+    # filtering on {'_cls': 'Human.GoodJedi'}
+    assert GoodJedi.objects().count() == 0
+
+    # Following has also no match
+    # because it is filtering on {'_cls': {'$in': ('Human', 'Human.GoodJedi', 'Human.BadSith')}}
+    # which has no match
+    assert Human.objects.count() == 0
+    assert Human.objects.first() is None
+
+    # If we bypass MongoEngine and make use of underlying driver (PyMongo)
+    # we can see that the documents are there
+    humans_coll = Human._get_collection()
+    assert humans_coll.count_documents({}) == 2
+    # print first document
+    print(humans_coll.find_one())
+    # {'_id': ObjectId('5fac4aaaf61d7fb06046e0f9'), '_cls': 'Human.Jedi', 'name': 'Darth Vader', 'dark_side': True, 'light_saber_color': 'red'}
+
+As you can see, first obvious problem is that we need to modify '_cls' values based on existing values of
+'dark_side' documents.
+
+.. code-block:: python
+
+    humans_coll = Human._get_collection()
+    old_class = 'Human.Jedi'
+    good_jedi_class = 'Human.GoodJedi'
+    bad_sith_class = 'Human.BadSith'
+    humans_coll.update_many({'_cls': old_class, 'dark_side': False}, {'$set': {'_cls': good_jedi_class}})
+    humans_coll.update_many({'_cls': old_class, 'dark_side': True}, {'$set': {'_cls': bad_sith_class}})
+
+Let's now check if querying improved in MongoEngine:
+
+.. code-block:: python
+
+    assert GoodJedi.objects().count() == 1  # Hoorah!
+    assert BadSith.objects().count() == 1   # Hoorah!
+    assert Human.objects.count() == 2       # Hoorah!
+
+    # let's now check that documents load correctly
+    jedi = GoodJedi.objects().first()
+    # raises FieldDoesNotExist: The fields "{'dark_side'}" do not exist on the document "Human.GoodJedi"
+
+In fact we only took care of renaming the _cls values but we havn't removed the 'dark_side' fields
+which does not exist anymore on the GoodJedi's and BadSith's models.
+Let's remove the field from the collections:
+
+.. code-block:: python
+
+    humans_coll = Human._get_collection()
+    humans_coll.update_many({}, {'$unset': {'dark_side': 1}})
+
+.. note:: We did this migration in 2 different steps for the sake of example but it could have been combined
+    with the migration of the _cls fields: ::
+
+        humans_coll.update_many(
+            {'_cls': old_class, 'dark_side': False},
+            {
+                '$set': {'_cls': good_jedi_class},
+                '$unset': {'dark_side': 1}
+            }
+        )
+
+
+And verify that the documents now load correctly:
+
+.. code-block:: python
+
+    jedi = GoodJedi.objects().first()
+    assert jedi.name == "Obi Wan Kenobi"
+
+    sith = BadSith.objects().first()
+    assert sith.name == "Darth Vader"
+
+
+An other way of dealing with this migration is to iterate over
+the documents and update/replace them one by one. This is way slower but
+it is often useful for complex migrations of Document models.
+
+.. code-block:: python
+
+    for doc in humans_coll.find():
+        if doc['_cls'] == 'Human.Jedi':
+            doc['_cls'] =  'Human.BadSith' if doc['dark_side'] else 'Human.GoodJedi'
+            doc.pop('dark_side')
+            humans_coll.replace_one({'_id': doc['_id']}, doc)
+
+.. warning:: Be aware of this `flaw <https://groups.google.com/g/mongodb-user/c/AFC1ia7MHzk>`_ if you modify documents while iterating
+
+Recommendations
+===============
+
+- Write migration scripts whenever you do changes to the model schemas
+- Using :class:`~mongoengine.DynamicDocument` or ``meta = {"strict": False}`` may help to avoid some migrations or to have the 2 versions of your application to co-exist.
+- Write post-processing checks to verify that migrations script worked. See below
+
+Post-processing checks
+======================
+
+The following recipe can be used to sanity check a Document collection after you applied migration.
+It does not make any assumption on what was migrated, it will fetch 1000 objects randomly and
+run some quick checks on the documents to make sure the document looks OK. As it is, it will fail
+on the first occurrence of an error but this is something that can be adapted based on your needs.
+
+.. code-block:: python
+
+    def get_random_oids(collection, sample_size):
+        pipeline = [{"$project": {'_id': 1}}, {"$sample": {"size": sample_size}}]
+        return [s['_id'] for s in collection.aggregate(pipeline)]
+
+    def get_random_documents(DocCls, sample_size):
+        doc_collection = DocCls._get_collection()
+        random_oids = get_random_oids(doc_collection, sample_size)
+        return DocCls.objects(id__in=random_oids)
+
+    def check_documents(DocCls, sample_size):
+        for doc in get_random_documents(DocCls, sample_size):
+            # general validation (types and values)
+            doc.validate()
+
+            # load all subfields,
+            # this may trigger additional queries if you have ReferenceFields
+            # so it may be slow
+            for field in doc._fields:
+                try:
+                    getattr(doc, field)
+                except Exception:
+                    LOG.warning(f"Could not load field {field} in Document {doc.id}")
+                    raise
+
+    check_documents(Human, sample_size=1000)

mongoengine/__init__.py

@@ -28,7 +28,7 @@ __all__ = (
 )
 
 
-VERSION = (0, 20, 0)
+VERSION = (0, 21, 0)
 
 
 def get_version():

mongoengine/base/document.py

@@ -101,12 +101,13 @@ class BaseDocument:
 
         self._dynamic_fields = SON()
 
-        # Assign default values to the instance.
-        for key, field in self._fields.items():
-            if self._db_field_map.get(key, key) in values:
+        # Assign default values for fields
+        # not set in the constructor
+        for field_name in self._fields:
+            if field_name in values:
                 continue
-            value = getattr(self, key, None)
-            setattr(self, key, value)
+            value = getattr(self, field_name, None)
+            setattr(self, field_name, value)
 
         if "_cls" not in values:
             self._cls = self._class_name
@@ -115,7 +116,6 @@ class BaseDocument:
         dynamic_data = {}
         FileField = _import_class("FileField")
         for key, value in values.items():
-            key = self._reverse_db_field_map.get(key, key)
             field = self._fields.get(key)
             if field or key in ("id", "pk", "_cls"):
                 if __auto_convert and value is not None:
@@ -750,7 +750,8 @@ class BaseDocument:
 
     @classmethod
     def _from_son(cls, son, _auto_dereference=True, created=False):
-        """Create an instance of a Document (subclass) from a PyMongo SON."""
+        """Create an instance of a Document (subclass) from a PyMongo SON (dict)
+        """
         if son and not isinstance(son, dict):
             raise ValueError(
                 "The source SON object needs to be of type 'dict' but a '%s' was found"
@@ -763,6 +764,8 @@ class BaseDocument:
 
         # Convert SON to a data dict, making sure each key is a string and
         # corresponds to the right db field.
+        # This is needed as _from_son is currently called both from BaseDocument.__init__
+        # and from EmbeddedDocumentField.to_python
         data = {}
         for key, value in son.items():
             key = str(key)

mongoengine/base/fields.py

@@ -1,5 +1,4 @@
 import operator
-import warnings
 import weakref
 
 from bson import DBRef, ObjectId, SON
@@ -16,11 +15,9 @@ __all__ = ("BaseField", "ComplexBaseField", "ObjectIdField", "GeoJsonBaseField")
 class BaseField:
     """A base class for fields in a MongoDB document. Instances of this class
     may be added to subclasses of `Document` to define a document's schema.
-
-    .. versionchanged:: 0.5 - added verbose and help text
     """
 
-    name = None
+    name = None  # set in TopLevelDocumentMetaclass
     _geo_index = False
     _auto_gen = False  # Call `generate` to generate a value
     _auto_dereference = True
@@ -265,11 +262,11 @@ class ComplexBaseField(BaseField):
     Allows for nesting of embedded documents inside complex types.
     Handles the lazy dereferencing of a queryset by lazily dereferencing all
     items in a list / dict rather than one at a time.
-
-    .. versionadded:: 0.5
     """
 
-    field = None
+    def __init__(self, field=None, **kwargs):
+        self.field = field
+        super().__init__(**kwargs)
 
     def __get__(self, instance, owner):
         """Descriptor to automatically dereference references."""
@@ -521,8 +518,6 @@ class ObjectIdField(BaseField):
 
 class GeoJsonBaseField(BaseField):
     """A geo json field storing a geojson style object.
-
-    .. versionadded:: 0.8
     """
 
     _geo_index = pymongo.GEOSPHERE

mongoengine/connection.py

@@ -74,8 +74,6 @@ def _get_connection_settings(
     : param kwargs: ad-hoc parameters to be passed into the pymongo driver,
         for example maxpoolsize, tz_aware, etc. See the documentation
         for pymongo's `MongoClient` for a full list.
-
-    .. versionchanged:: 0.10.6 - added mongomock support
     """
     conn_settings = {
         "name": name or db or DEFAULT_DATABASE_NAME,
@@ -201,8 +199,6 @@ def register_connection(
     : param kwargs: ad-hoc parameters to be passed into the pymongo driver,
         for example maxpoolsize, tz_aware, etc. See the documentation
         for pymongo's `MongoClient` for a full list.
-
-    .. versionchanged:: 0.10.6 - added mongomock support
     """
     conn_settings = _get_connection_settings(
         db=db,
@@ -386,8 +382,6 @@ def connect(db=None, alias=DEFAULT_CONNECTION_NAME, **kwargs):
 
     See the docstring for `register_connection` for more details about all
     supported kwargs.
-
-    .. versionchanged:: 0.6 - added multiple database support.
     """
     if alias in _connections:
         prev_conn_setting = _connection_settings[alias]

mongoengine/document.py

@@ -1,5 +1,4 @@
 import re
-import warnings
 
 from bson.dbref import DBRef
 import pymongo
@@ -367,15 +366,6 @@ class Document(BaseDocument, metaclass=TopLevelDocumentMetaclass):
             meta['cascade'] = True.  Also you can pass different kwargs to
             the cascade save using cascade_kwargs which overwrites the
             existing kwargs with custom values.
-        .. versionchanged:: 0.8.5
-            Optional save_condition that only overwrites existing documents
-            if the condition is satisfied in the current db record.
-        .. versionchanged:: 0.10
-            :class:`OperationError` exception raised if save_condition fails.
-        .. versionchanged:: 0.10.1
-            :class: save_condition failure now raises a `SaveConditionError`
-        .. versionchanged:: 0.10.7
-            Add signal_kwargs argument
         """
         signal_kwargs = signal_kwargs or {}
 
@@ -630,9 +620,6 @@ class Document(BaseDocument, metaclass=TopLevelDocumentMetaclass):
             For example, ``save(..., w: 2, fsync: True)`` will
             wait until at least two servers have recorded the write and
             will force an fsync on the primary server.
-
-        .. versionchanged:: 0.10.7
-            Add signal_kwargs argument
         """
         signal_kwargs = signal_kwargs or {}
         signals.pre_delete.send(self.__class__, document=self, **signal_kwargs)
@@ -714,8 +701,6 @@ class Document(BaseDocument, metaclass=TopLevelDocumentMetaclass):
     def select_related(self, max_depth=1):
         """Handles dereferencing of :class:`~bson.dbref.DBRef` objects to
         a maximum depth in order to cut down the number queries to mongodb.
-
-        .. versionadded:: 0.5
         """
         DeReference = _import_class("DeReference")
         DeReference()([self], max_depth + 1)
@@ -726,10 +711,6 @@ class Document(BaseDocument, metaclass=TopLevelDocumentMetaclass):
 
         :param fields: (optional) args list of fields to reload
         :param max_depth: (optional) depth of dereferencing to follow
-
-        .. versionadded:: 0.1.2
-        .. versionchanged:: 0.6  Now chainable
-        .. versionchanged:: 0.9  Can provide specific fields to reload
         """
         max_depth = 1
         if fields and isinstance(fields[0], int):
@@ -831,9 +812,6 @@ class Document(BaseDocument, metaclass=TopLevelDocumentMetaclass):
 
         Raises :class:`OperationError` if the document has no collection set
         (i.g. if it is `abstract`)
-
-        .. versionchanged:: 0.10.7
-            :class:`OperationError` exception raised if no collection available
         """
         coll_name = cls._get_collection_name()
         if not coll_name:
@@ -1088,8 +1066,6 @@ class MapReduceDocument:
                 an ``ObjectId`` found in the given ``collection``,
                 the object can be accessed via the ``object`` property.
     :param value: The result(s) for this key.
-
-    .. versionadded:: 0.3
     """
 
     def __init__(self, document, collection, key, value):

mongoengine/fields.py

@@ -36,7 +36,6 @@ from mongoengine.common import _import_class
 from mongoengine.connection import DEFAULT_CONNECTION_NAME, get_db
 from mongoengine.document import Document, EmbeddedDocument
 from mongoengine.errors import DoesNotExist, InvalidQueryError, ValidationError
-from mongoengine.mongodb_support import MONGODB_36, get_mongodb_version
 from mongoengine.queryset import DO_NOTHING
 from mongoengine.queryset.base import BaseQuerySet
 from mongoengine.queryset.transform import STRING_OPERATORS
@@ -101,6 +100,12 @@ class StringField(BaseField):
     """A unicode string field."""
 
     def __init__(self, regex=None, max_length=None, min_length=None, **kwargs):
+        """
+        :param regex: (optional) A string pattern that will be applied during validation
+        :param max_length: (optional) A max length that will be applied during validation
+        :param min_length: (optional) A min length that will be applied during validation
+        :param kwargs: Keyword arguments passed into the parent :class:`~mongoengine.BaseField`
+        """
         self.regex = re.compile(regex) if regex else None
         self.max_length = max_length
         self.min_length = min_length
@@ -156,10 +161,7 @@ class StringField(BaseField):
 
 
 class URLField(StringField):
-    """A field that validates input as an URL.
-
-    .. versionadded:: 0.3
-    """
+    """A field that validates input as an URL."""
 
     _URL_REGEX = LazyRegexCompiler(
         r"^(?:[a-z0-9\.\-]*)://"  # scheme is validated separately
@@ -174,6 +176,11 @@ class URLField(StringField):
     _URL_SCHEMES = ["http", "https", "ftp", "ftps"]
 
     def __init__(self, url_regex=None, schemes=None, **kwargs):
+        """
+        :param url_regex: (optional) Overwrite the default regex used for validation
+        :param schemes: (optional) Overwrite the default URL schemes that are allowed
+        :param kwargs: Keyword arguments passed into the parent :class:`~mongoengine.StringField`
+        """
         self.url_regex = url_regex or self._URL_REGEX
         self.schemes = schemes or self._URL_SCHEMES
         super().__init__(**kwargs)
@@ -192,7 +199,6 @@ class URLField(StringField):
 class EmailField(StringField):
     """A field that validates input as an email address.
 
-    .. versionadded:: 0.4
     """
 
     USER_REGEX = LazyRegexCompiler(
@@ -229,16 +235,11 @@ class EmailField(StringField):
         *args,
         **kwargs
     ):
-        """Initialize the EmailField.
-
-        Args:
-            domain_whitelist (list) - list of otherwise invalid domain
-                                      names which you'd like to support.
-            allow_utf8_user (bool) - if True, the user part of the email
-                                     address can contain UTF8 characters.
-                                     False by default.
-            allow_ip_domain (bool) - if True, the domain part of the email
-                                     can be a valid IPv4 or IPv6 address.
+        """
+        :param domain_whitelist: (optional) list of valid domain names applied during validation
+        :param allow_utf8_user: Allow user part of the email to contain utf8 char
+        :param allow_ip_domain: Allow domain part of the email to be an IPv4 or IPv6 address
+        :param kwargs: Keyword arguments passed into the parent :class:`~mongoengine.StringField`
         """
         self.domain_whitelist = domain_whitelist or []
         self.allow_utf8_user = allow_utf8_user
@@ -310,6 +311,11 @@ class IntField(BaseField):
     """32-bit integer field."""
 
     def __init__(self, min_value=None, max_value=None, **kwargs):
+        """
+        :param min_value: (optional) A min value that will be applied during validation
+        :param max_value: (optional) A max value that will be applied during validation
+        :param kwargs: Keyword arguments passed into the parent :class:`~mongoengine.BaseField`
+        """
         self.min_value, self.max_value = min_value, max_value
         super().__init__(**kwargs)
 
@@ -343,6 +349,11 @@ class LongField(BaseField):
     """64-bit integer field. (Equivalent to IntField since the support to Python2 was dropped)"""
 
     def __init__(self, min_value=None, max_value=None, **kwargs):
+        """
+        :param min_value: (optional) A min value that will be applied during validation
+        :param max_value: (optional) A max value that will be applied during validation
+        :param kwargs: Keyword arguments passed into the parent :class:`~mongoengine.BaseField`
+        """
         self.min_value, self.max_value = min_value, max_value
         super().__init__(**kwargs)
 
@@ -379,6 +390,11 @@ class FloatField(BaseField):
     """Floating point number field."""
 
     def __init__(self, min_value=None, max_value=None, **kwargs):
+        """
+        :param min_value: (optional) A min value that will be applied during validation
+        :param max_value: (optional) A max value that will be applied during validation
+        :param kwargs: Keyword arguments passed into the parent :class:`~mongoengine.BaseField`
+        """
         self.min_value, self.max_value = min_value, max_value
         super().__init__(**kwargs)
 
@@ -415,9 +431,6 @@ class FloatField(BaseField):
 class DecimalField(BaseField):
     """Fixed-point decimal number field. Stores the value as a float by default unless `force_string` is used.
     If using floats, beware of Decimal to float conversion (potential precision loss)
-
-    .. versionchanged:: 0.8
-    .. versionadded:: 0.3
     """
 
     def __init__(
@@ -430,8 +443,8 @@ class DecimalField(BaseField):
         **kwargs
     ):
         """
-        :param min_value: Validation rule for the minimum acceptable value.
-        :param max_value: Validation rule for the maximum acceptable value.
+        :param min_value: (optional) A min value that will be applied during validation
+        :param max_value: (optional) A max value that will be applied during validation
         :param force_string: Store the value as a string (instead of a float).
          Be aware that this affects query sorting and operation like lte, gte (as string comparison is applied)
          and some query operator won't work (e.g. inc, dec)
@@ -448,7 +461,7 @@ class DecimalField(BaseField):
             - decimal.ROUND_05UP (away from zero if last digit after rounding towards zero would have been 0 or 5; otherwise towards zero)
 
             Defaults to: ``decimal.ROUND_HALF_UP``
-
+        :param kwargs: Keyword arguments passed into the parent :class:`~mongoengine.BaseField`
         """
         self.min_value = min_value
         self.max_value = max_value
@@ -498,10 +511,7 @@ class DecimalField(BaseField):
 
 
 class BooleanField(BaseField):
-    """Boolean field type.
-
-    .. versionadded:: 0.1.2
-    """
+    """Boolean field type."""
 
     def to_python(self, value):
         try:
@@ -546,12 +556,13 @@ class DateTimeField(BaseField):
         if callable(value):
             return value()
 
-        if not isinstance(value, str):
+        if isinstance(value, str):
+            return self._parse_datetime(value)
+        else:
             return None
 
-        return self._parse_datetime(value)
-
-    def _parse_datetime(self, value):
+    @staticmethod
+    def _parse_datetime(value):
         # Attempt to parse a datetime from a string
         value = value.strip()
         if not value:
@@ -627,13 +638,12 @@ class ComplexDateTimeField(StringField):
     keyword when initializing the field.
 
     Note: To default the field to the current datetime, use: DateTimeField(default=datetime.utcnow)
-
-    .. versionadded:: 0.5
     """
 
     def __init__(self, separator=",", **kwargs):
         """
         :param separator: Allows to customize the separator used for storage (default ``,``)
+        :param kwargs: Keyword arguments passed into the parent :class:`~mongoengine.StringField`
         """
         self.separator = separator
         self.format = separator.join(["%Y", "%m", "%d", "%H", "%M", "%S", "%f"])
@@ -913,10 +923,9 @@ class ListField(ComplexBaseField):
     """
 
     def __init__(self, field=None, max_length=None, **kwargs):
-        self.field = field
         self.max_length = max_length
         kwargs.setdefault("default", lambda: [])
-        super().__init__(**kwargs)
+        super().__init__(field=field, **kwargs)
 
     def __get__(self, instance, owner):
         if instance is None:
@@ -975,16 +984,13 @@ class EmbeddedDocumentListField(ListField):
     .. note::
         The only valid list values are subclasses of
         :class:`~mongoengine.EmbeddedDocument`.
-
-    .. versionadded:: 0.9
     """
 
     def __init__(self, document_type, **kwargs):
         """
         :param document_type: The type of
          :class:`~mongoengine.EmbeddedDocument` the list will hold.
-        :param kwargs: Keyword arguments passed directly into the parent
-         :class:`~mongoengine.ListField`.
+        :param kwargs: Keyword arguments passed into the parent :class:`~mongoengine.ListField`
         """
         super().__init__(field=EmbeddedDocumentField(document_type), **kwargs)
 
@@ -999,19 +1005,11 @@ class SortedListField(ListField):
         save the whole list then other processes trying to save the whole list
         as well could overwrite changes.  The safest way to append to a list is
         to perform a push operation.
-
-    .. versionadded:: 0.4
-    .. versionchanged:: 0.6 - added reverse keyword
     """
 
-    _ordering = None
-    _order_reverse = False
-
     def __init__(self, field, **kwargs):
-        if "ordering" in kwargs.keys():
-            self._ordering = kwargs.pop("ordering")
-        if "reverse" in kwargs.keys():
-            self._order_reverse = kwargs.pop("reverse")
+        self._ordering = kwargs.pop("ordering", None)
+        self._order_reverse = kwargs.pop("reverse", False)
         super().__init__(field, **kwargs)
 
     def to_mongo(self, value, use_db_field=True, fields=None):
@@ -1058,17 +1056,13 @@ class DictField(ComplexBaseField):
 
     .. note::
         Required means it cannot be empty - as the default for DictFields is {}
-
-    .. versionadded:: 0.3
-    .. versionchanged:: 0.5 - Can now handle complex / varying types of data
     """
 
     def __init__(self, field=None, *args, **kwargs):
-        self.field = field
         self._auto_dereference = False
 
         kwargs.setdefault("default", lambda: {})
-        super().__init__(*args, **kwargs)
+        super().__init__(*args, field=field, **kwargs)
 
     def validate(self, value):
         """Make sure that a list of valid fields is being used."""
@@ -1124,8 +1118,6 @@ class MapField(DictField):
     """A field that maps a name to a specified field type. Similar to
     a DictField, except the 'value' of each item must match the specified
     field type.
-
-    .. versionadded:: 0.5
     """
 
     def __init__(self, field=None, *args, **kwargs):
@@ -1173,8 +1165,6 @@ class ReferenceField(BaseField):
             org = ReferenceField('Org', reverse_delete_rule=CASCADE)
 
         User.register_delete_rule(Org, 'owner', DENY)
-
-    .. versionchanged:: 0.5 added `reverse_delete_rule`
     """
 
     def __init__(
@@ -1182,10 +1172,12 @@ class ReferenceField(BaseField):
     ):
         """Initialises the Reference Field.
 
+        :param document_type: The type of Document that will be referenced
         :param dbref:  Store the reference as :class:`~pymongo.dbref.DBRef`
           or as the :class:`~pymongo.objectid.ObjectId`.id .
         :param reverse_delete_rule: Determines what to do when the referring
           object is deleted
+        :param kwargs: Keyword arguments passed into the parent :class:`~mongoengine.BaseField`
 
         .. note ::
             A reference to an abstract document type is always stored as a
@@ -1307,17 +1299,16 @@ class ReferenceField(BaseField):
 
 
 class CachedReferenceField(BaseField):
-    """
-    A referencefield with cache fields to purpose pseudo-joins
-
-    .. versionadded:: 0.9
+    """A referencefield with cache fields to purpose pseudo-joins
     """
 
     def __init__(self, document_type, fields=None, auto_sync=True, **kwargs):
         """Initialises the Cached Reference Field.
 
+        :param document_type: The type of Document that will be referenced
         :param fields:  A list of fields to be cached in document
-        :param auto_sync: if True documents are auto updated.
+        :param auto_sync: if True documents are auto updated
+        :param kwargs: Keyword arguments passed into the parent :class:`~mongoengine.BaseField`
         """
         if fields is None:
             fields = []
@@ -1485,8 +1476,6 @@ class GenericReferenceField(BaseField):
           it.
 
         * You can use the choices param to limit the acceptable Document types
-
-    .. versionadded:: 0.3
     """
 
     def __init__(self, *args, **kwargs):
@@ -1692,10 +1681,6 @@ class GridFSError(Exception):
 
 class GridFSProxy:
     """Proxy object to handle writing and reading of files to and from GridFS
-
-    .. versionadded:: 0.4
-    .. versionchanged:: 0.5 - added optional size param to read
-    .. versionchanged:: 0.6 - added collection name param
     """
 
     _fs = None
@@ -1859,10 +1844,6 @@ class GridFSProxy:
 
 class FileField(BaseField):
     """A GridFS storage field.
-
-    .. versionadded:: 0.4
-    .. versionchanged:: 0.5 added optional size param for read
-    .. versionchanged:: 0.6 added db_alias for multidb support
     """
 
     proxy_class = GridFSProxy
@@ -1945,11 +1926,7 @@ class FileField(BaseField):
 
 
 class ImageGridFsProxy(GridFSProxy):
-    """
-    Proxy for ImageField
-
-    versionadded: 0.6
-    """
+    """Proxy for ImageField"""
 
     def put(self, file_obj, **kwargs):
         """
@@ -2083,8 +2060,6 @@ class ImageField(FileField):
     :param size: max size to store images, provided as (width, height, force)
         if larger, it will be automatically resized (ex: size=(800, 600, True))
     :param thumbnail_size: size to generate a thumbnail, provided as (width, height, force)
-
-    .. versionadded:: 0.6
     """
 
     proxy_class = ImageGridFsProxy
@@ -2132,9 +2107,6 @@ class SequenceField(BaseField):
         In case the counter is defined in the abstract document, it will be
         common to all inherited documents and the default sequence name will
         be the class name of the abstract document.
-
-    .. versionadded:: 0.5
-    .. versionchanged:: 0.8 added `value_decorator`
     """
 
     _auto_gen = True
@@ -2248,8 +2220,6 @@ class SequenceField(BaseField):
 
 class UUIDField(BaseField):
     """A UUID field.
-
-    .. versionadded:: 0.6
     """
 
     _binary = None
@@ -2259,9 +2229,6 @@ class UUIDField(BaseField):
         Store UUID data in the database
 
         :param binary: if False store as a string.
-
-        .. versionchanged:: 0.8.0
-        .. versionchanged:: 0.6.19
         """
         self._binary = binary
         super().__init__(**kwargs)
@@ -2306,8 +2273,6 @@ class GeoPointField(BaseField):
         representing a geo point. It admits 2d indexes but not "2dsphere" indexes
         in MongoDB > 2.4 which are more natural for modeling geospatial points.
         See :ref:`geospatial-indexes`
-
-    .. versionadded:: 0.4
     """
 
     _geo_index = pymongo.GEO2D
@@ -2339,8 +2304,6 @@ class PointField(GeoJsonBaseField):
     to set the value.
 
     Requires mongodb >= 2.4
-
-    .. versionadded:: 0.8
     """
 
     _type = "Point"
@@ -2359,8 +2322,6 @@ class LineStringField(GeoJsonBaseField):
     You can either pass a dict with the full information or a list of points.
 
     Requires mongodb >= 2.4
-
-    .. versionadded:: 0.8
     """
 
     _type = "LineString"
@@ -2382,8 +2343,6 @@ class PolygonField(GeoJsonBaseField):
     holes.
 
     Requires mongodb >= 2.4
-
-    .. versionadded:: 0.8
     """
 
     _type = "Polygon"
@@ -2403,8 +2362,6 @@ class MultiPointField(GeoJsonBaseField):
     to set the value.
 
     Requires mongodb >= 2.6
-
-    .. versionadded:: 0.9
     """
 
     _type = "MultiPoint"
@@ -2424,8 +2381,6 @@ class MultiLineStringField(GeoJsonBaseField):
     You can either pass a dict with the full information or a list of points.
 
     Requires mongodb >= 2.6
-
-    .. versionadded:: 0.9
     """
 
     _type = "MultiLineString"
@@ -2452,8 +2407,6 @@ class MultiPolygonField(GeoJsonBaseField):
     of Polygons.
 
     Requires mongodb >= 2.6
-
-    .. versionadded:: 0.9
     """
 
     _type = "MultiPolygon"
@@ -2466,8 +2419,6 @@ class LazyReferenceField(BaseField):
     Instead, access will return a :class:`~mongoengine.base.LazyReference` class
     instance, allowing access to `pk` or manual dereference by using
     ``fetch()`` method.
-
-    .. versionadded:: 0.15
     """
 
     def __init__(
@@ -2570,6 +2521,7 @@ class LazyReferenceField(BaseField):
         if not isinstance(value, (DBRef, Document, EmbeddedDocument)):
             collection = self.document_type._get_collection_name()
             value = DBRef(collection, self.document_type.id.to_python(value))
+            value = self.build_lazyref(value)
         return value
 
     def validate(self, value):
@@ -2630,8 +2582,6 @@ class GenericLazyReferenceField(GenericReferenceField):
           it.
 
         * You can use the choices param to limit the acceptable Document types
-
-    .. versionadded:: 0.15
     """
 
     def __init__(self, *args, **kwargs):

mongoengine/queryset/base.py

@@ -256,8 +256,6 @@ class BaseQuerySet:
         `DocumentName.MultipleObjectsReturned` exception if multiple results
         and :class:`~mongoengine.queryset.DoesNotExist` or
         `DocumentName.DoesNotExist` if no results are found.
-
-        .. versionadded:: 0.3
         """
         queryset = self.clone()
         queryset = queryset.order_by().limit(2)
@@ -282,8 +280,6 @@ class BaseQuerySet:
 
     def create(self, **kwargs):
         """Create new object. Returns the saved object instance.
-
-        .. versionadded:: 0.4
         """
         return self._document(**kwargs).save(force_insert=True)
 
@@ -316,10 +312,6 @@ class BaseQuerySet:
 
         By default returns document instances, set ``load_bulk`` to False to
         return just ``ObjectIds``
-
-        .. versionadded:: 0.5
-        .. versionchanged:: 0.10.7
-            Add signal_kwargs argument
         """
         Document = _import_class("Document")
 
@@ -550,8 +542,6 @@ class BaseQuerySet:
         :param update: Django-style update keyword arguments
 
         :returns the number of updated documents (unless ``full_result`` is True)
-
-        .. versionadded:: 0.2
         """
         if not update and not upsert:
             raise OperationError("No update parameters, would remove data")
@@ -603,8 +593,6 @@ class BaseQuerySet:
         :param update: Django-style update keyword arguments
 
         :returns the new or overwritten document
-
-        .. versionadded:: 0.10.2
         """
 
         atomic_update = self.update(
@@ -638,7 +626,6 @@ class BaseQuerySet:
         :param update: Django-style update keyword arguments
             full_result
         :returns the number of updated documents (unless ``full_result`` is True)
-        .. versionadded:: 0.2
         """
         return self.update(
             upsert=upsert,
@@ -670,8 +657,6 @@ class BaseQuerySet:
         :param new: return updated rather than original document
             (default ``False``)
         :param update: Django-style update keyword arguments
-
-        .. versionadded:: 0.9
         """
 
         if remove and new:
@@ -727,8 +712,6 @@ class BaseQuerySet:
         `None` if no document exists with that id.
 
         :param object_id: the value for the id of the document to look up
-
-        .. versionchanged:: 0.6 Raises InvalidQueryError if filter has been set
         """
         queryset = self.clone()
         if not queryset._query_obj.empty:
@@ -742,8 +725,6 @@ class BaseQuerySet:
         :param object_ids: a list or tuple of ObjectId's
         :rtype: dict of ObjectId's as keys and collection-specific
                 Document subclasses as values.
-
-        .. versionadded:: 0.3
         """
         doc_map = {}
 
@@ -785,8 +766,6 @@ class BaseQuerySet:
         evaluated against if you are using more than one database.
 
         :param alias: The database alias
-
-        .. versionadded:: 0.9
         """
 
         with switch_db(self._document, alias) as cls:
@@ -848,8 +827,6 @@ class BaseQuerySet:
         """Handles dereferencing of :class:`~bson.dbref.DBRef` objects or
         :class:`~bson.object_id.ObjectId` a maximum depth in order to cut down
         the number queries to mongodb.
-
-        .. versionadded:: 0.5
         """
         # Make select related work the same for querysets
         max_depth += 1
@@ -898,8 +875,6 @@ class BaseQuerySet:
 
         Hinting will not do anything if the corresponding index does not exist.
         The last hint applied to this cursor takes precedence over all others.
-
-        .. versionadded:: 0.5
         """
         queryset = self.clone()
         queryset._hint = index
@@ -961,10 +936,6 @@ class BaseQuerySet:
 
         .. note:: This is a command and won't take ordering or limit into
            account.
-
-        .. versionadded:: 0.4
-        .. versionchanged:: 0.5 - Fixed handling references
-        .. versionchanged:: 0.6 - Improved db_field refrence handling
         """
         queryset = self.clone()
 
@@ -1028,9 +999,6 @@ class BaseQuerySet:
         field filters.
 
         :param fields: fields to include
-
-        .. versionadded:: 0.3
-        .. versionchanged:: 0.5 - Added subfield support
         """
         fields = {f: QueryFieldList.ONLY for f in fields}
         return self.fields(True, **fields)
@@ -1049,8 +1017,6 @@ class BaseQuerySet:
         field filters.
 
         :param fields: fields to exclude
-
-        .. versionadded:: 0.5
         """
         fields = {f: QueryFieldList.EXCLUDE for f in fields}
         return self.fields(**fields)
@@ -1077,8 +1043,6 @@ class BaseQuerySet:
 
         :param kwargs: A set of keyword arguments identifying what to
             include, exclude, or slice.
-
-        .. versionadded:: 0.5
         """
 
         # Check for an operator and transform to mongo-style if there is
@@ -1120,8 +1084,6 @@ class BaseQuerySet:
         .exclude(). ::
 
             post = BlogPost.objects.exclude('comments').all_fields()
-
-        .. versionadded:: 0.5
         """
         queryset = self.clone()
         queryset._loaded_fields = QueryFieldList(
@@ -1194,9 +1156,6 @@ class BaseQuerySet:
         """Enable or disable snapshot mode when querying.
 
         :param enabled: whether or not snapshot mode is enabled
-
-        ..versionchanged:: 0.5 - made chainable
-        .. deprecated:: Ignored with PyMongo 3+
         """
         msg = "snapshot is deprecated as it has no impact when using PyMongo 3+."
         warnings.warn(msg, DeprecationWarning)
@@ -1208,8 +1167,6 @@ class BaseQuerySet:
         """Enable or disable the default mongod timeout when querying. (no_cursor_timeout option)
 
         :param enabled: whether or not the timeout is used
-
-        ..versionchanged:: 0.5 - made chainable
         """
         queryset = self.clone()
         queryset._timeout = enabled
@@ -1308,7 +1265,6 @@ class BaseQuerySet:
             parameter will be removed shortly
         :param kwargs: (optional) kwargs dictionary to be passed to pymongo's aggregate call
             See https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.aggregate
-        .. versionadded:: 0.9
         """
         using_deprecated_interface = isinstance(pipeline, dict) or bool(suppl_pipeline)
         user_pipeline = [pipeline] if isinstance(pipeline, dict) else list(pipeline)
@@ -1380,12 +1336,6 @@ class BaseQuerySet:
             Map/Reduce changed in server version **>= 1.7.4**. The PyMongo
             :meth:`~pymongo.collection.Collection.map_reduce` helper requires
             PyMongo version **>= 1.11**.
-
-        .. versionchanged:: 0.5
-           - removed ``keep_temp`` keyword argument, which was only relevant
-             for MongoDB server versions older than 1.7.4
-
-        .. versionadded:: 0.3
         """
         queryset = self.clone()
 
@@ -1522,8 +1472,6 @@ class BaseQuerySet:
         .. note:: When using this mode of query, the database will call your
                   function, or evaluate your predicate clause, for each object
                   in the collection.
-
-        .. versionadded:: 0.5
         """
         queryset = self.clone()
         where_clause = queryset._sub_js_fields(where_clause)
@@ -1600,9 +1548,6 @@ class BaseQuerySet:
         :param field: the field to use
         :param normalize: normalize the results so they add to 1.0
         :param map_reduce: Use map_reduce over exec_js
-
-        .. versionchanged:: 0.5 defaults to map_reduce and can handle embedded
-                            document lookups
         """
         if map_reduce:
             return self._item_frequencies_map_reduce(field, normalize=normalize)
@@ -1632,8 +1577,6 @@ class BaseQuerySet:
 
     def rewind(self):
         """Rewind the cursor to its unevaluated state.
-
-        .. versionadded:: 0.3
         """
         self._iter = False
         self._cursor.rewind()

mongoengine/queryset/queryset.py

@@ -151,8 +151,6 @@ class QuerySet(BaseQuerySet):
 
     def no_cache(self):
         """Convert to a non-caching queryset
-
-        .. versionadded:: 0.8.3 Convert to non caching queryset
         """
         if self._result_cache is not None:
             raise OperationError("QuerySet already cached")
@@ -165,15 +163,11 @@ class QuerySetNoCache(BaseQuerySet):
 
     def cache(self):
         """Convert to a caching queryset
-
-        .. versionadded:: 0.8.3 Convert to caching queryset
         """
         return self._clone_into(QuerySet(self._document, self._collection))
 
     def __repr__(self):
         """Provides the string representation of the QuerySet
-
-        .. versionchanged:: 0.6.13 Now doesnt modify the cursor
         """
         if self._iter:
             return ".. queryset mid-iteration .."

setup.py

@@ -115,7 +115,7 @@ extra_opts = {
         "pytest-cov",
         "coverage<5.0",  # recent coverage switched to sqlite format for the .coverage file which isn't handled properly by coveralls
         "blinker",
-        "Pillow>=2.0.0, <7.0.0",  # 7.0.0 dropped Python2 support
+        "Pillow>=7.0.0",
     ],
 }
 

tests/document/test_instance.py

@@ -3822,5 +3822,95 @@ class ObjectKeyTestCase(MongoDBTestCase):
         assert book._object_key == {"pk": book.pk, "author__name": "Author"}
 
 
+class DBFieldMappingTest(MongoDBTestCase):
+    def setUp(self):
+        class Fields(object):
+            w1 = BooleanField(db_field="w2")
+
+            x1 = BooleanField(db_field="x2")
+            x2 = BooleanField(db_field="x3")
+
+            y1 = BooleanField(db_field="y0")
+            y2 = BooleanField(db_field="y1")
+
+            z1 = BooleanField(db_field="z2")
+            z2 = BooleanField(db_field="z1")
+
+        class Doc(Fields, Document):
+            pass
+
+        class DynDoc(Fields, DynamicDocument):
+            pass
+
+        self.Doc = Doc
+        self.DynDoc = DynDoc
+
+    def tearDown(self):
+        for collection in list_collection_names(self.db):
+            self.db.drop_collection(collection)
+
+    def test_setting_fields_in_constructor_of_strict_doc_uses_model_names(self):
+        doc = self.Doc(z1=True, z2=False)
+        assert doc.z1 is True
+        assert doc.z2 is False
+
+    def test_setting_fields_in_constructor_of_dyn_doc_uses_model_names(self):
+        doc = self.DynDoc(z1=True, z2=False)
+        assert doc.z1 is True
+        assert doc.z2 is False
+
+    def test_setting_unknown_field_in_constructor_of_dyn_doc_does_not_overwrite_model_fields(
+        self,
+    ):
+        doc = self.DynDoc(w2=True)
+        assert doc.w1 is None
+        assert doc.w2 is True
+
+    def test_unknown_fields_of_strict_doc_do_not_overwrite_dbfields_1(self):
+        doc = self.Doc()
+        doc.w2 = True
+        doc.x3 = True
+        doc.y0 = True
+        doc.save()
+        reloaded = self.Doc.objects.get(id=doc.id)
+        assert reloaded.w1 is None
+        assert reloaded.x1 is None
+        assert reloaded.x2 is None
+        assert reloaded.y1 is None
+        assert reloaded.y2 is None
+
+    def test_dbfields_are_loaded_to_the_right_modelfield_for_strict_doc_2(self):
+        doc = self.Doc()
+        doc.x2 = True
+        doc.y2 = True
+        doc.z2 = True
+        doc.save()
+        reloaded = self.Doc.objects.get(id=doc.id)
+        assert (
+            reloaded.x1,
+            reloaded.x2,
+            reloaded.y1,
+            reloaded.y2,
+            reloaded.z1,
+            reloaded.z2,
+        ) == (doc.x1, doc.x2, doc.y1, doc.y2, doc.z1, doc.z2)
+
+    def test_dbfields_are_loaded_to_the_right_modelfield_for_dyn_doc_2(self):
+        doc = self.DynDoc()
+        doc.x2 = True
+        doc.y2 = True
+        doc.z2 = True
+        doc.save()
+        reloaded = self.DynDoc.objects.get(id=doc.id)
+        assert (
+            reloaded.x1,
+            reloaded.x2,
+            reloaded.y1,
+            reloaded.y2,
+            reloaded.z1,
+            reloaded.z2,
+        ) == (doc.x1, doc.x2, doc.y1, doc.y2, doc.z1, doc.z2)
+
+
 if __name__ == "__main__":
     unittest.main()

tests/fields/test_fields.py

@@ -2272,6 +2272,13 @@ class TestField(MongoDBTestCase):
         with pytest.raises(FieldDoesNotExist):
             Doc(bar="test")
 
+    def test_undefined_field_works_no_confusion_with_db_field(self):
+        class Doc(Document):
+            foo = StringField(db_field="bar")
+
+        with pytest.raises(FieldDoesNotExist):
+            Doc(bar="test")
+
 
 class TestEmbeddedDocumentListField(MongoDBTestCase):
     def setUp(self):

tests/fields/test_file_field.py

@@ -429,7 +429,7 @@ class TestFileField(MongoDBTestCase):
     @require_pil
     def test_image_field_resize(self):
         class TestImage(Document):
-            image = ImageField(size=(185, 37))
+            image = ImageField(size=(185, 37, True))
 
         TestImage.drop_collection()
 
@@ -471,7 +471,7 @@ class TestFileField(MongoDBTestCase):
     @require_pil
     def test_image_field_thumbnail(self):
         class TestImage(Document):
-            image = ImageField(thumbnail_size=(92, 18))
+            image = ImageField(thumbnail_size=(92, 18, True))
 
         TestImage.drop_collection()
 

tests/fields/test_lazy_reference_field.py

@@ -3,6 +3,7 @@ import pytest
 
 from mongoengine import *
 from mongoengine.base import LazyReference
+from mongoengine.context_managers import query_counter
 
 from tests.utils import MongoDBTestCase
 
@@ -330,6 +331,50 @@ class TestLazyReferenceField(MongoDBTestCase):
         occ.in_embedded.in_list = [animal1.id, animal2.id]
         check_fields_type(occ)
 
+    def test_lazy_reference_embedded_dereferencing(self):
+        # Test case for #2375
+
+        # -- Test documents
+
+        class Author(Document):
+            name = StringField()
+
+        class AuthorReference(EmbeddedDocument):
+            author = LazyReferenceField(Author)
+
+        class Book(Document):
+            authors = EmbeddedDocumentListField(AuthorReference)
+
+        # -- Cleanup
+
+        Author.drop_collection()
+        Book.drop_collection()
+
+        # -- Create test data
+
+        author_1 = Author(name="A1").save()
+        author_2 = Author(name="A2").save()
+        author_3 = Author(name="A3").save()
+        book = Book(
+            authors=[
+                AuthorReference(author=author_1),
+                AuthorReference(author=author_2),
+                AuthorReference(author=author_3),
+            ]
+        ).save()
+
+        with query_counter() as qc:
+            book = Book.objects.first()
+            # Accessing the list must not trigger dereferencing.
+            book.authors
+            assert qc == 1
+
+        for ref in book.authors:
+            with pytest.raises(AttributeError):
+                ref["author"].name
+            assert isinstance(ref.author, LazyReference)
+            assert isinstance(ref.author.id, ObjectId)
+
 
 class TestGenericLazyReferenceField(MongoDBTestCase):
     def test_generic_lazy_reference_simple(self):