pub/mongoengine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'master' into fix_2484

Browse Source
This commit is contained in:
Jan Stein 2021-03-25 15:27:48 +01:00
commit da173cf0e2
14 changed files with 186 additions and 100 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
.github/workflows/github-actions.yml vendored
View File

@ -11,9 +11,10 @@ on:
tags:
- 'v[0-9]+\.[0-9]+\.[0-9]+*'
env:
MONGODB_3_4: 3.4.19
MONGODB_3_6: 3.6.13
MONGODB_4_0: 4.0.13
MONGODB_3_6: 3.6.14
MONGODB_4_0: 4.0.23
MONGODB_4_2: 4.2
MONGODB_4_4: 4.4
PYMONGO_3_4: 3.4
PYMONGO_3_6: 3.6
@ -47,14 +48,14 @@ jobs:
MONGODB: [$MONGODB_4_0]
PYMONGO: [$PYMONGO_3_11]
include:
- python-version: 3.7
MONGODB: $MONGODB_3_4
PYMONGO: $PYMONGO_3_6
- python-version: 3.7
MONGODB: $MONGODB_3_6
PYMONGO: $PYMONGO_3_9
- python-version: 3.7
MONGODB: $MONGODB_3_6
MONGODB: $MONGODB_4_2
PYMONGO: $PYMONGO_3_6
- python-version: 3.7
MONGODB: $MONGODB_4_4
PYMONGO: $PYMONGO_3_11
steps:
- uses: actions/checkout@v2

12
.github/workflows/install_mongo.sh vendored
View File

@ -2,7 +2,17 @@
MONGODB=$1
# Mongo > 4.0 follows different name convention for download links
mongo_build=mongodb-linux-x86_64-${MONGODB}
if [[ "$MONGODB" == *"4.2"* ]]; then
mongo_build=mongodb-linux-x86_64-ubuntu1804-v${MONGODB}-latest
elif [[ "$MONGODB" == *"4.4"* ]]; then
mongo_build=mongodb-linux-x86_64-ubuntu1804-v${MONGODB}-latest
fi
wget http://fastdl.mongodb.org/linux/$mongo_build.tgz
tar xzf $mongo_build.tgz
${PWD}/$mongo_build/bin/mongod --version
mongodb_dir=$(find ${PWD}/ -type d -name "mongodb-linux-x86_64*")
$mongodb_dir/bin/mongod --version

3
.github/workflows/start_mongo.sh vendored
View File

@ -2,7 +2,8 @@
MONGODB=$1
mongodb_dir=${PWD}/mongodb-linux-x86_64-${MONGODB}
mongodb_dir=$(find ${PWD}/ -type d -name "mongodb-linux-x86_64*")
mkdir $mongodb_dir/data
$mongodb_dir/bin/mongod --dbpath $mongodb_dir/data --logpath $mongodb_dir/mongodb.log --fork
mongo --eval 'db.version();' # Make sure mongo is awake

9
docs/changelog.rst
View File

@ -1,4 +1,5 @@
=========
Changelog
=========
@ -6,8 +7,14 @@ Changelog
Development
===========
- (Fill this out as you fix issues and develop your features).
- Bugfix: manually setting SequenceField in DynamicDocument doesn't increment the counter #2471
- Bug fix: ignore LazyReferenceFields when clearing _changed_fields #2484
- Improve connection doc #2481
Changes in 0.23.0
=================
- Bugfix: manually setting SequenceField in DynamicDocument doesn't increment the counter #2471
- Add MongoDB 4.2 and 4.4 to CI
- Add support for allowDiskUse on querysets #2468
Changes in 0.22.1
=================

88
docs/guide/connecting.rst
View File

@ -5,7 +5,7 @@ Connecting to MongoDB
=====================
Connections in MongoEngine are registered globally and are identified with aliases.
If no `alias` is provided during the connection, it will use "default" as alias.
If no ``alias`` is provided during the connection, it will use "default" as alias.
To connect to a running instance of :program:`mongod`, use the :func:`~mongoengine.connect`
function. The first argument is the name of the database to connect to::
@ -14,27 +14,66 @@ function. The first argument is the name of the database to connect to::
connect('project1')
By default, MongoEngine assumes that the :program:`mongod` instance is running
on **localhost** on port **27017**. If MongoDB is running elsewhere, you should
provide the :attr:`host` and :attr:`port` arguments to
:func:`~mongoengine.connect`::
on **localhost** on port **27017**.
connect('project1', host='192.168.1.35', port=12345)
If MongoDB is running elsewhere, you need to provide details on how to connect. There are two ways of
doing this. Using a connection string in URI format (**this is the preferred method**) or individual attributes
provided as keyword arguments.
Connect with URI string
=======================
When using a connection string in URI format you should specify the connection details
as the :attr:`host` to :func:`~mongoengine.connect`. In a web application context for instance, the URI
is typically read from the config file::
connect(host="mongodb://127.0.0.1:27017/my_db")
If the database requires authentication, you can specify it in the
URI. As each database can have its own users configured, you need to tell MongoDB
where to look for the user you are working with, that's what the ``?authSource=admin`` bit
of the MongoDB connection string is for::
# Connects to 'my_db' database by authenticating
# with given credentials against the 'admin' database (by default as authSource isn't provided)
connect(host="mongodb://my_user:my_password@127.0.0.1:27017/my_db")
# Equivalent to previous connection but explicitly states that
# it should use admin as the authentication source database
connect(host="mongodb://my_user:my_password@hostname:port/my_db?authSource=admin")
# Connects to 'my_db' database by authenticating
# with given credentials against that same database
connect(host="mongodb://my_user:my_password@127.0.0.1:27017/my_db?authSource=my_db")
The URI string can also be used to configure advanced parameters like ssl, replicaSet, etc. For more
information or example about URI string, you can refer to the `official doc <https://docs.mongodb.com/manual/reference/connection-string/>`_::
connect(host="mongodb://my_user:my_password@127.0.0.1:27017/my_db?authSource=admin&ssl=true&replicaSet=globaldb")
.. note:: URI containing SRV records (e.g "mongodb+srv://server.example.com/") can be used as well
Connect with keyword attributes
===============================
The second option for specifying the connection details is to provide the information as keyword
attributes to :func:`~mongoengine.connect`::
connect('my_db', host='127.0.0.1', port=27017)
If the database requires authentication, :attr:`username`, :attr:`password`
and :attr:`authentication_source` arguments should be provided::
connect('project1', username='webapp', password='pwd123', authentication_source='admin')
connect('my_db', username='my_user', password='my_password', authentication_source='admin')
URI style connections are also supported -- just supply the URI as
the :attr:`host` to
:func:`~mongoengine.connect`::
connect('project1', host='mongodb://localhost/database_name')
.. note:: URI containing SRV records (e.g mongodb+srv://server.example.com/) can be used as well as the :attr:`host`
The set of attributes that :func:`~mongoengine.connect` recognizes includes but is not limited to:
:attr:`host`, :attr:`port`, :attr:`read_preference`, :attr:`username`, :attr:`password`, :attr:`authentication_source`, :attr:`authentication_mechanism`,
:attr:`replicaset`, :attr:`tls`, etc. Most of the parameters accepted by `pymongo.MongoClient <https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient>`_
can be used with :func:`~mongoengine.connect` and will simply be forwarded when instantiating the `pymongo.MongoClient`.
.. note:: Database, username and password from URI string overrides
corresponding parameters in :func:`~mongoengine.connect`: ::
corresponding parameters in :func:`~mongoengine.connect`, this should
obviously be avoided: ::
connect(
db='test',
@ -43,28 +82,19 @@ the :attr:`host` to
host='mongodb://admin:qwerty@localhost/production'
)
will establish connection to ``production`` database using
``admin`` username and ``12345`` password.
will establish connection to ``production`` database using ``admin`` username and ``qwerty`` password.
.. note:: Calling :func:`~mongoengine.connect` without argument will establish
a connection to the "test" database by default
Replica Sets
============
Read Preferences
================
MongoEngine supports connecting to replica sets::
from mongoengine import connect
# Regular connect
connect('dbname', replicaset='rs-name')
# MongoDB URI-style connect
connect(host='mongodb://localhost/dbname?replicaSet=rs-name')
Read preferences are supported through the connection or via individual
As stated above, Read preferences are supported through the connection but also via individual
queries by passing the read_preference ::
from pymongo import ReadPreference
Bar.objects().read_preference(ReadPreference.PRIMARY)
Bar.objects(read_preference=ReadPreference.PRIMARY)

6
docs/guide/defining-documents.rst
View File

@ -290,12 +290,12 @@ as the constructor's argument::
content = StringField()
.. _one-to-many-with-listfields:
.. _many-to-many-with-listfields:
One to Many with ListFields
Many to Many with ListFields
'''''''''''''''''''''''''''
If you are implementing a one to many relationship via a list of references,
If you are implementing a many to many relationship via a list of references,
then the references are stored as DBRefs and to query you need to pass an
instance of the object to the query::

2
mongoengine/__init__.py
View File

@ -28,7 +28,7 @@ __all__ = (
)
VERSION = (0, 22, 1)
VERSION = (0, 23, 0)
def get_version():

25
mongoengine/base/fields.py
View File

@ -267,6 +267,17 @@ class ComplexBaseField(BaseField):
self.field = field
super().__init__(**kwargs)
@staticmethod
def _lazy_load_refs(instance, name, ref_values, *, max_depth):
_dereference = _import_class("DeReference")()
documents = _dereference(
ref_values,
max_depth=max_depth,
instance=instance,
name=name,
)
return documents
def __get__(self, instance, owner):
"""Descriptor to automatically dereference references."""
if instance is None:
@ -284,19 +295,15 @@ class ComplexBaseField(BaseField):
or isinstance(self.field, (GenericReferenceField, ReferenceField))
)
_dereference = _import_class("DeReference")()
if (
instance._initialised
and dereference
and instance._data.get(self.name)
and not getattr(instance._data[self.name], "_dereferenced", False)
):
instance._data[self.name] = _dereference(
instance._data.get(self.name),
max_depth=1,
instance=instance,
name=self.name,
ref_values = instance._data.get(self.name)
instance._data[self.name] = self._lazy_load_refs(
ref_values=ref_values, instance=instance, name=self.name, max_depth=1
)
if hasattr(instance._data[self.name], "_dereferenced"):
instance._data[self.name]._dereferenced = True
@ -322,7 +329,9 @@ class ComplexBaseField(BaseField):
and isinstance(value, (BaseList, BaseDict))
and not value._dereferenced
):
value = _dereference(value, max_depth=1, instance=instance, name=self.name)
value = self._lazy_load_refs(
ref_values=value, instance=instance, name=self.name, max_depth=1
)
value._dereferenced = True
instance._data[self.name] = value

64
mongoengine/fields.py
View File

@ -915,7 +915,7 @@ class ListField(ComplexBaseField):
"""A list field that wraps a standard field, allowing multiple instances
of the field to be used as a list in the database.
If using with ReferenceFields see: :ref:`one-to-many-with-listfields`
If using with ReferenceFields see: :ref:`many-to-many-with-listfields`
.. note::
Required means it cannot be empty - as the default for ListFields is []
@ -1194,6 +1194,14 @@ class ReferenceField(BaseField):
self.document_type_obj = get_document(self.document_type_obj)
return self.document_type_obj
@staticmethod
def _lazy_load_ref(ref_cls, dbref):
dereferenced_son = ref_cls._get_db().dereference(dbref)
if dereferenced_son is None:
raise DoesNotExist(f"Trying to dereference unknown document {dbref}")
return ref_cls._from_son(dereferenced_son)
def __get__(self, instance, owner):
"""Descriptor to allow lazy dereferencing."""
if instance is None:
@ -1201,20 +1209,17 @@ class ReferenceField(BaseField):
return self
# Get value from document instance if available
value = instance._data.get(self.name)
ref_value = instance._data.get(self.name)
auto_dereference = instance._fields[self.name]._auto_dereference
# Dereference DBRefs
if auto_dereference and isinstance(value, DBRef):
if hasattr(value, "cls"):
if auto_dereference and isinstance(ref_value, DBRef):
if hasattr(ref_value, "cls"):
# Dereference using the class type specified in the reference
cls = get_document(value.cls)
cls = get_document(ref_value.cls)
else:
cls = self.document_type
dereferenced = cls._get_db().dereference(value)
if dereferenced is None:
raise DoesNotExist("Trying to dereference unknown document %s" % value)
else:
instance._data[self.name] = cls._from_son(dereferenced)
instance._data[self.name] = self._lazy_load_ref(cls, ref_value)
return super().__get__(instance, owner)
@ -1353,6 +1358,14 @@ class CachedReferenceField(BaseField):
self.document_type_obj = get_document(self.document_type_obj)
return self.document_type_obj
@staticmethod
def _lazy_load_ref(ref_cls, dbref):
dereferenced_son = ref_cls._get_db().dereference(dbref)
if dereferenced_son is None:
raise DoesNotExist(f"Trying to dereference unknown document {dbref}")
return ref_cls._from_son(dereferenced_son)
def __get__(self, instance, owner):
if instance is None:
# Document class being used rather than a document object
@ -1364,11 +1377,7 @@ class CachedReferenceField(BaseField):
# Dereference DBRefs
if auto_dereference and isinstance(value, DBRef):
dereferenced = self.document_type._get_db().dereference(value)
if dereferenced is None:
raise DoesNotExist("Trying to dereference unknown document %s" % value)
else:
instance._data[self.name] = self.document_type._from_son(dereferenced)
instance._data[self.name] = self._lazy_load_ref(self.document_type, value)
return super().__get__(instance, owner)
@ -1493,6 +1502,14 @@ class GenericReferenceField(BaseField):
value = value._class_name
super()._validate_choices(value)
@staticmethod
def _lazy_load_ref(ref_cls, dbref):
dereferenced_son = ref_cls._get_db().dereference(dbref)
if dereferenced_son is None:
raise DoesNotExist(f"Trying to dereference unknown document {dbref}")
return ref_cls._from_son(dereferenced_son)
def __get__(self, instance, owner):
if instance is None:
return self
@ -1500,12 +1517,9 @@ class GenericReferenceField(BaseField):
value = instance._data.get(self.name)
auto_dereference = instance._fields[self.name]._auto_dereference
if auto_dereference and isinstance(value, (dict, SON)):
dereferenced = self.dereference(value)
if dereferenced is None:
raise DoesNotExist("Trying to dereference unknown document %s" % value)
else:
instance._data[self.name] = dereferenced
if auto_dereference and isinstance(value, dict):
doc_cls = get_document(value["_cls"])
instance._data[self.name] = self._lazy_load_ref(doc_cls, value["_ref"])
return super().__get__(instance, owner)
@ -1524,14 +1538,6 @@ class GenericReferenceField(BaseField):
" saved to the database"
)
def dereference(self, value):
doc_cls = get_document(value["_cls"])
reference = value["_ref"]
doc = doc_cls._get_db().dereference(reference)
if doc is not None:
doc = doc_cls._from_son(doc)
return doc
def to_mongo(self, document):
if document is None:
return None

2
mongoengine/mongodb_support.py
View File

@ -8,6 +8,8 @@ from mongoengine.connection import get_connection
# get_mongodb_version()
MONGODB_34 = (3, 4)
MONGODB_36 = (3, 6)
MONGODB_42 = (4, 2)
MONGODB_44 = (4, 4)
def get_mongodb_version():

9
mongoengine/queryset/base.py
View File

@ -1355,21 +1355,18 @@ class BaseQuerySet:
MapReduceDocument = _import_class("MapReduceDocument")
if not hasattr(self._collection, "map_reduce"):
raise NotImplementedError("Requires MongoDB >= 1.7.1")
map_f_scope = {}
if isinstance(map_f, Code):
map_f_scope = map_f.scope
map_f = str(map_f)
map_f = Code(queryset._sub_js_fields(map_f), map_f_scope)
map_f = Code(queryset._sub_js_fields(map_f), map_f_scope or None)
reduce_f_scope = {}
if isinstance(reduce_f, Code):
reduce_f_scope = reduce_f.scope
reduce_f = str(reduce_f)
reduce_f_code = queryset._sub_js_fields(reduce_f)
reduce_f = Code(reduce_f_code, reduce_f_scope)
reduce_f = Code(reduce_f_code, reduce_f_scope or None)
mr_args = {"query": queryset._query}
@ -1379,7 +1376,7 @@ class BaseQuerySet:
finalize_f_scope = finalize_f.scope
finalize_f = str(finalize_f)
finalize_f_code = queryset._sub_js_fields(finalize_f)
finalize_f = Code(finalize_f_code, finalize_f_scope)
finalize_f = Code(finalize_f_code, finalize_f_scope or None)
mr_args["finalize"] = finalize_f
if scope:

5
tests/document/test_indexes.py
View File

@ -7,6 +7,7 @@ import pytest
from mongoengine import *
from mongoengine.connection import get_db
from mongoengine.mongodb_support import MONGODB_42, get_mongodb_version
class TestIndexes(unittest.TestCase):
@ -452,9 +453,11 @@ class TestIndexes(unittest.TestCase):
.get("stage")
== "IXSCAN"
)
mongo_db = get_mongodb_version()
PROJECTION_STR = "PROJECTION" if mongo_db < MONGODB_42 else "PROJECTION_COVERED"
assert (
query_plan.get("queryPlanner").get("winningPlan").get("stage")
== "PROJECTION"
== PROJECTION_STR
)
query_plan = Test.objects(a=1).explain()

42
tests/queryset/test_queryset.py
View File

@ -21,7 +21,10 @@ from mongoengine.queryset import (
QuerySetManager,
queryset_manager,
)
from tests.utils import requires_mongodb_gte_44
from tests.utils import (
requires_mongodb_gte_44,
requires_mongodb_lt_42,
)
class db_ops_tracker(query_counter):
@ -1490,6 +1493,7 @@ class TestQueryset(unittest.TestCase):
BlogPost.drop_collection()
@requires_mongodb_lt_42
def test_exec_js_query(self):
"""Ensure that queries are properly formed for use in exec_js."""
@ -1527,6 +1531,7 @@ class TestQueryset(unittest.TestCase):
BlogPost.drop_collection()
@requires_mongodb_lt_42
def test_exec_js_field_sub(self):
"""Ensure that field substitutions occur properly in exec_js functions."""
@ -2660,6 +2665,8 @@ class TestQueryset(unittest.TestCase):
title = StringField(primary_key=True)
tags = ListField(StringField())
BlogPost.drop_collection()
post1 = BlogPost(title="Post #1", tags=["mongodb", "mongoengine"])
post2 = BlogPost(title="Post #2", tags=["django", "mongodb"])
post3 = BlogPost(title="Post #3", tags=["hitchcock films"])
@ -2688,12 +2695,15 @@ class TestQueryset(unittest.TestCase):
}
"""
results = BlogPost.objects.map_reduce(map_f, reduce_f, "myresults")
results = BlogPost.objects.order_by("_id").map_reduce(
map_f, reduce_f, "myresults2"
)
results = list(results)
assert results[0].object == post1
assert results[1].object == post2
assert results[2].object == post3
assert len(results) == 3
assert results[0].object.id == post1.id
assert results[1].object.id == post2.id
assert results[2].object.id == post3.id
BlogPost.drop_collection()
@ -2701,7 +2711,6 @@ class TestQueryset(unittest.TestCase):
"""
Test map/reduce custom output
"""
register_connection("test2", "mongoenginetest2")
class Family(Document):
id = IntField(primary_key=True)
@ -2774,6 +2783,7 @@ class TestQueryset(unittest.TestCase):
family.persons.push(person);
family.totalAge += person.age;
});
family.persons.sort((a, b) => (a.age > b.age))
}
});
@ -2802,10 +2812,10 @@ class TestQueryset(unittest.TestCase):
"_id": 1,
"value": {
"persons": [
{"age": 21, "name": "Wilson Jr"},
{"age": 45, "name": "Wilson Father"},
{"age": 40, "name": "Eliana Costa"},
{"age": 17, "name": "Tayza Mariana"},
{"age": 21, "name": "Wilson Jr"},
{"age": 40, "name": "Eliana Costa"},
{"age": 45, "name": "Wilson Father"},
],
"totalAge": 123,
},
@ -2815,9 +2825,9 @@ class TestQueryset(unittest.TestCase):
"_id": 2,
"value": {
"persons": [
{"age": 10, "name": "Igor Gabriel"},
{"age": 16, "name": "Isabella Luanna"},
{"age": 36, "name": "Sandra Mara"},
{"age": 10, "name": "Igor Gabriel"},
],
"totalAge": 62,
},
@ -2827,8 +2837,8 @@ class TestQueryset(unittest.TestCase):
"_id": 3,
"value": {
"persons": [
{"age": 30, "name": "Arthur WA"},
{"age": 25, "name": "Paula Leonel"},
{"age": 30, "name": "Arthur WA"},
],
"totalAge": 55,
},
@ -3109,6 +3119,7 @@ class TestQueryset(unittest.TestCase):
freq = Person.objects.item_frequencies("city", normalize=True, map_reduce=True)
assert freq == {"CRB": 0.5, None: 0.5}
@requires_mongodb_lt_42
def test_item_frequencies_with_null_embedded(self):
class Data(EmbeddedDocument):
name = StringField()
@ -3137,6 +3148,7 @@ class TestQueryset(unittest.TestCase):
ot = Person.objects.item_frequencies("extra.tag", map_reduce=True)
assert ot == {None: 1.0, "friend": 1.0}
@requires_mongodb_lt_42
def test_item_frequencies_with_0_values(self):
class Test(Document):
val = IntField()
@ -3151,6 +3163,7 @@ class TestQueryset(unittest.TestCase):
ot = Test.objects.item_frequencies("val", map_reduce=False)
assert ot == {0: 1}
@requires_mongodb_lt_42
def test_item_frequencies_with_False_values(self):
class Test(Document):
val = BooleanField()
@ -3165,6 +3178,7 @@ class TestQueryset(unittest.TestCase):
ot = Test.objects.item_frequencies("val", map_reduce=False)
assert ot == {False: 1}
@requires_mongodb_lt_42
def test_item_frequencies_normalize(self):
class Test(Document):
val = IntField()
@ -3551,7 +3565,8 @@ class TestQueryset(unittest.TestCase):
Book.objects.create(title="The Stories", authors=[mark_twain, john_tolkien])
authors = Book.objects.distinct("authors")
assert authors == [mark_twain, john_tolkien]
authors_names = {author.name for author in authors}
assert authors_names == {mark_twain.name, john_tolkien.name}
def test_distinct_ListField_EmbeddedDocumentField_EmbeddedDocumentField(self):
class Continent(EmbeddedDocument):
@ -3588,7 +3603,8 @@ class TestQueryset(unittest.TestCase):
assert country_list == [scotland, tibet]
continent_list = Book.objects.distinct("authors.country.continent")
assert continent_list == [europe, asia]
continent_list_names = {c.continent_name for c in continent_list}
assert continent_list_names == {europe.continent_name, asia.continent_name}
def test_distinct_ListField_ReferenceField(self):
class Bar(Document):

4
tests/utils.py
View File

@ -34,6 +34,10 @@ def get_as_pymongo(doc):
return doc.__class__.objects.as_pymongo().get(id=doc.id)
def requires_mongodb_lt_42(func):
return _decorated_with_ver_requirement(func, (4, 2), oper=operator.lt)
def requires_mongodb_gte_44(func):
return _decorated_with_ver_requirement(func, (4, 4), oper=operator.ge)