55 Commits

Author SHA1 Message Date
Vasilios Syrakis
a5effb219a Release 1.2.4 (#71)
Co-authored-by: nat <n@natn.me>
2020-05-26 22:17:55 +02:00
Bouke Versteegh
2d57f0d122 Merge pull request #67 from danielgtaylor/nat-n-patch-1
Enforce utf-8 for reading the readme in setup.py
2020-05-25 21:57:12 +02:00
nat
a68505b80e Enforce utf-8 for reading the readme
Fixes failing installation issue #66
2020-05-25 17:53:13 +02:00
nat
2f9497e064 Merge pull request #55 from boukeversteegh/pr/xfail-tests
Add intentionally failing test-cases for unimplemented bug-fixes
2020-05-25 09:54:26 +02:00
boukeversteegh
33964b883e Do not use mutable defaults 2020-05-25 00:35:43 +02:00
boukeversteegh
ec7574086d Add xfail test-case to for future circular dependency scenario 2020-05-24 20:35:10 +02:00
boukeversteegh
8a42027bc9 Improve failing test-case for issue #64 2020-05-24 20:33:48 +02:00
boukeversteegh
71737cf696 Test case for issue #63 2020-05-24 20:29:32 +02:00
boukeversteegh
659ddd9c44 Working test case for oneof 2020-05-24 20:29:19 +02:00
boukeversteegh
5b6997870a Test case for issue #61 2020-05-24 20:27:12 +02:00
boukeversteegh
cdf7645722 Test case for issue #60 2020-05-24 20:26:47 +02:00
boukeversteegh
ca20069ca3 Test case for issue #59 2020-05-24 20:26:13 +02:00
boukeversteegh
59a4a7da43 Test case for issue #58 2020-05-24 20:25:29 +02:00
boukeversteegh
15af4367e5 Test case for issue #57 2020-05-24 20:24:55 +02:00
boukeversteegh
ec5683e572 Test Service instantiation as part of standard test-case 2020-05-24 20:02:41 +02:00
boukeversteegh
20150fdcf3 Cleanup 2020-05-24 19:58:49 +02:00
boukeversteegh
d11b7d04c5 Document XFAIL tests 2020-05-24 19:58:35 +02:00
boukeversteegh
e2d35f4696 Support xfail on test-case level, support running tests on subsets. 2020-05-24 19:58:06 +02:00
boukeversteegh
c3f08b9ef2 Clear output directories before generating python files 2020-05-24 19:54:53 +02:00
boukeversteegh
24d44898f4 Only import reference module when needed. Some reference modules generate bad imports and cannot be loaded. 2020-05-24 19:53:14 +02:00
boukeversteegh
074448c996 Restore accidentally removed binary equality test 2020-05-24 19:52:14 +02:00
nat
0fe557bd3c Merge pull request #52 from nat-n/fix_type_imports
Only import types from grpclib when type checking
2020-05-24 19:09:08 +02:00
nat
1a87ea43a1 Merge pull request #40 from boukeversteegh/pr/wrapper-as-output
Support using Google's wrapper types as RPC output values
2020-05-24 19:06:30 +02:00
andrei
983e0895a2 Fix services using non-pythonified field names 2020-05-24 18:46:36 +02:00
nat
4a2baf3f0a Merge pull request #46 from jameslan/perf/class-cache
Improve performance of serialize/deserialize by caching type information of fields in class
2020-05-24 18:38:32 +02:00
boukeversteegh
8f0caf1db2 Read desired wrapper type directly from wrapper definition 2020-05-24 14:50:56 +02:00
boukeversteegh
c50d9e2fdc Add test for generating embedded wellknown types in outputs. 2020-05-24 14:48:39 +02:00
boukeversteegh
35548cb43e Test all supported wrapper types. Add xfail test for unwrapping the value 2020-05-24 12:34:37 +02:00
boukeversteegh
b711d1e11f Merge remote-tracking branch 'daniel/master' into pr/wrapper-as-output 2020-05-24 10:41:40 +02:00
James Lan
917de09bb6 Replace extra decorator with property and lazy initialization so that it is backward compatible. 2020-05-23 17:36:29 -07:00
James Lan
1f7f39049e Cache resolved classes for fields, so that there's no new data classes generated while deserializing. 2020-05-23 17:36:29 -07:00
James Lan
3d001a2a1a Store the class metadata of fields in the class, to improve preformance
Cached data include,
- lookup table between groups and fields of "oneof" fields
- default value creator of each field
- type hint of each field
2020-05-23 17:36:29 -07:00
James Lan
de61ddab21 Add option to repeatly execute betterproto operations in test, to evaluate performance 2020-05-23 17:36:29 -07:00
Nat Noordanus
5e2d9febea Blacken 2020-05-23 23:37:22 +02:00
nat
f6af077ffe Merge pull request #51 from boukeversteegh/pr/refactor-tests
Reorganize tests and add some extra documentation.
2020-05-22 22:32:37 +02:00
boukeversteegh
92088ebda8 Cleanup 2020-05-22 21:18:44 +02:00
boukeversteegh
c3e3837f71 More concise whitelist logic 2020-05-22 21:11:23 +02:00
boukeversteegh
6bd9c7835c Fix docs 2020-05-22 21:08:08 +02:00
boukeversteegh
6ec902c1b5 Fix generate noargs. Sorted iteration. 2020-05-22 21:03:45 +02:00
boukeversteegh
960dba2ae8 Renamed docs for standard tests 2020-05-22 20:58:53 +02:00
boukeversteegh
4b4bdefb6f Add explicit test for casing rules 2020-05-22 20:58:31 +02:00
boukeversteegh
dfa0a56b39 Simplify standard tests by using 1 json per case. 2020-05-22 20:58:14 +02:00
boukeversteegh
dd4873dfba Re-introducing whitelisting argument to generate.py 2020-05-22 20:51:22 +02:00
Nat Noordanus
91f586f7d7 Apply black formatting 2020-05-22 18:46:43 +02:00
Nat Noordanus
33fb83faad Only import types from grpclib when type checking 2020-05-22 18:41:29 +02:00
boukeversteegh
77c04414f5 Update readme, add docs for standard tests 2020-05-22 16:36:43 +02:00
boukeversteegh
6969ff7ff6 Add another missing gitignored file, and remove gitignore filter for tests/ 2020-05-22 15:34:25 +02:00
boukeversteegh
13e08fdaa8 Add missing file, ignore output files 2020-05-22 15:05:52 +02:00
boukeversteegh
6775632f77 Undo unintentional pipfile update 2020-05-22 13:03:52 +02:00
boukeversteegh
b12f1e4e61 Organize test-cases into folders, extract compatibility test into proper test, support adding test-case specific tests 2020-05-22 12:54:01 +02:00
Bouke Versteegh
7e9ba0866c cleanup 2020-05-21 22:55:26 +02:00
nat
3546f55146 Merge pull request #32 from nat-n/improve_stub
Add ability to provide metadata, timeout & deadline args to requests
2020-05-21 10:11:45 +02:00
boukeversteegh
499489f1d3 Support using Google's wrapper types as RPC output values 2020-05-10 16:36:29 +02:00
Nat Noordanus
5759e323bd Add ability to provide metadata, timeout & deadline args to requests
This is an enhancement of the ServiceStub abstract class that makes
it more useful by making it possible to pass all arguments supported
by the underlying grpclib request function.

It extends to the existing high level API by allowing values to be
set on the stub instance, and the low level API by allowing values
to be set per call.
2020-04-12 22:23:10 +02:00
Nat Noordanus
c762c9c549 Add test for generated service stub
- Create one simple test for generated Service stubs in preparation
for making more changes in this area.
- Add dev dependency on pytest-asyncio in order to use ChannelFor
from grpclib.testing more easily.
- Create a new example proto containing a minimal rpc example.
2020-04-12 19:37:39 +02:00
94 changed files with 1722 additions and 222 deletions

7
.gitignore vendored
View File

@@ -4,12 +4,9 @@
.pytest_cache
.python-version
build/
betterproto/tests/*.bin
betterproto/tests/*_pb2.py
betterproto/tests/*.py
!betterproto/tests/generate.py
!betterproto/tests/test_*.py
betterproto/tests/output_*
**/__pycache__
dist
**/*.egg-info
output
.idea

View File

@@ -5,6 +5,16 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [1.2.4] - 2020-04-26
- Enforce utf-8 for reading the readme in setup.py [#67](https://github.com/danielgtaylor/python-betterproto/pull/67)
- Only import types from grpclib when type checking [#52](https://github.com/danielgtaylor/python-betterproto/pull/52)
- Improve performance of serialize/deserialize by caching type information of fields in class [#46](https://github.com/danielgtaylor/python-betterproto/pull/46)
- Support using Google's wrapper types as RPC output values [#40](https://github.com/danielgtaylor/python-betterproto/pull/40)
- Fixes issue where protoc did not recognize plugin.py as win32 application [#38](https://github.com/danielgtaylor/python-betterproto/pull/38)
- Fix services using non-pythonified field names [#34](https://github.com/danielgtaylor/python-betterproto/pull/34)
- Add ability to provide metadata, timeout & deadline args to requests [#32](https://github.com/danielgtaylor/python-betterproto/pull/32)
## [1.2.3] - 2020-04-15
- Exclude empty lists from `to_dict` by default [#16](https://github.com/danielgtaylor/python-betterproto/pull/16)
@@ -44,7 +54,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Initial release
[unreleased]: https://github.com/danielgtaylor/python-betterproto/compare/v1.2.3...HEAD
[1.2.4]: https://github.com/danielgtaylor/python-betterproto/compare/v1.2.3...v1.2.4
[1.2.3]: https://github.com/danielgtaylor/python-betterproto/compare/v1.2.2...v1.2.3
[1.2.2]: https://github.com/danielgtaylor/python-betterproto/compare/v1.2.1...v1.2.2
[1.2.1]: https://github.com/danielgtaylor/python-betterproto/compare/v1.2.0...v1.2.1

View File

@@ -8,6 +8,7 @@ flake8 = "*"
mypy = "*"
isort = "*"
pytest = "*"
pytest-asyncio = "*"
rope = "*"
v = {editable = true,version = "*"}

464
Pipfile.lock generated Normal file
View File

@@ -0,0 +1,464 @@
{
"_meta": {
"hash": {
"sha256": "b8fc738d4e14598e36269ce0d849489f95562ba047e5663caca9ac02550893ef"
},
"pipfile-spec": 6,
"requires": {
"python_version": "3.6"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.org/simple",
"verify_ssl": true
}
]
},
"default": {
"appdirs": {
"hashes": [
"sha256:9e5896d1372858f8dd3344faf4e5014d21849c756c8d5701f78f8a103b372d92",
"sha256:d8b24664561d0d34ddfaec54636d502d7cea6e29c3eaf68f3df6180863e2166e"
],
"version": "==1.4.3"
},
"attrs": {
"hashes": [
"sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c",
"sha256:f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72"
],
"version": "==19.3.0"
},
"backports-datetime-fromisoformat": {
"hashes": [
"sha256:9577a2a9486cd7383a5f58b23bb8e81cf0821dbbc0eb7c87d3fa198c1df40f5c"
],
"index": "pypi",
"version": "==1.0.0"
},
"black": {
"hashes": [
"sha256:1b30e59be925fafc1ee4565e5e08abef6b03fe455102883820fe5ee2e4734e0b",
"sha256:c2edb73a08e9e0e6f65a0e6af18b059b8b1cdd5bef997d7a0b181df93dc81539"
],
"index": "pypi",
"version": "==19.10b0"
},
"click": {
"hashes": [
"sha256:8a18b4ea89d8820c5d0c7da8a64b2c324b4dabb695804dbfea19b9be9d88c0cc",
"sha256:e345d143d80bf5ee7534056164e5e112ea5e22716bbb1ce727941f4c8b471b9a"
],
"version": "==7.1.1"
},
"dataclasses": {
"hashes": [
"sha256:454a69d788c7fda44efd71e259be79577822f5e3f53f029a22d08004e951dc9f",
"sha256:6988bd2b895eef432d562370bb707d540f32f7360ab13da45340101bc2307d84"
],
"index": "pypi",
"version": "==0.6"
},
"grpclib": {
"hashes": [
"sha256:b27d56c987b89023d5640fe9668943e49b46703fc85d8182a58c9f3b19120cdc"
],
"index": "pypi",
"version": "==0.3.2rc1"
},
"h2": {
"hashes": [
"sha256:61e0f6601fa709f35cdb730863b4e5ec7ad449792add80d1410d4174ed139af5",
"sha256:875f41ebd6f2c44781259005b157faed1a5031df3ae5aa7bcb4628a6c0782f14"
],
"version": "==3.2.0"
},
"hpack": {
"hashes": [
"sha256:0edd79eda27a53ba5be2dfabf3b15780928a0dff6eb0c60a3d6767720e970c89",
"sha256:8eec9c1f4bfae3408a3f30500261f7e6a65912dc138526ea054f9ad98892e9d2"
],
"version": "==3.0.0"
},
"hyperframe": {
"hashes": [
"sha256:5187962cb16dcc078f23cb5a4b110098d546c3f41ff2d4038a9896893bbd0b40",
"sha256:a9f5c17f2cc3c719b917c4f33ed1c61bd1f8dfac4b1bd23b7c80b3400971b41f"
],
"version": "==5.2.0"
},
"jinja2": {
"hashes": [
"sha256:c10142f819c2d22bdcd17548c46fa9b77cf4fda45097854c689666bf425e7484",
"sha256:c922560ac46888d47384de1dbdc3daaa2ea993af4b26a436dec31fa2c19ec668"
],
"index": "pypi",
"version": "==3.0.0a1"
},
"markupsafe": {
"hashes": [
"sha256:06358015a4dee8ee23ae426bf885616ab3963622defd829eb45b44e3dee3515f",
"sha256:0b0c4fc852c5f02c6277ef3b33d23fcbe89b1b227460423e3335374da046b6db",
"sha256:267677fc42afed5094fc5ea1c4236bbe4b6a00fe4b08e93451e65ae9048139c7",
"sha256:303cb70893e2c345588fb5d5b86e0ca369f9bb56942f03064c5e3e75fa7a238a",
"sha256:3c9b624a0d9ed5a5093ac4edc4e823e6b125441e60ef35d36e6f4a6fdacd5054",
"sha256:42033e14cae1f6c86fc0c3e90d04d08ce73ac8e46ba420a0d22d545c2abd4977",
"sha256:4e4a99b6af7bdc0856b50020c095848ec050356a001e1f751510aef6ab14d0e0",
"sha256:4eb07faad54bb07427d848f31030a65a49ebb0cec0b30674f91cf1ddd456bfe4",
"sha256:63a7161cd8c2bc563feeda45df62f42c860dd0675e2b8da2667f25bb3c95eaba",
"sha256:68e0fd039b68d2945b4beb947d4023ca7f8e95b708031c345762efba214ea761",
"sha256:8092a63397025c2f655acd42784b2a1528339b90b987beb9253f22e8cdbb36c3",
"sha256:841218860683c0f2223e24756843d84cc49cccdae6765e04962607754a52d3e0",
"sha256:94076b2314bd2f6cfae508ad65b4d493e3a58a50112b7a2cbb6287bdbc404ae8",
"sha256:9d22aff1c5322e402adfb3ce40839a5056c353e711c033798cf4f02eb9f5124d",
"sha256:b0e4584f62b3e5f5c1a7bcefd2b52f236505e6ef032cc508caa4f4c8dc8d3af1",
"sha256:b1163ffc1384d242964426a8164da12dbcdbc0de18ea36e2c34b898ed38c3b45",
"sha256:beac28ed60c8e838301226a7a85841d0af2068eba2dcb1a58c2d32d6c05e440e",
"sha256:c29f096ce79c03054a1101d6e5fe6bf04b0bb489165d5e0e9653fb4fe8048ee1",
"sha256:c58779966d53e5f14ba393d64e2402a7926601d1ac8adeb4e83893def79d0428",
"sha256:cfe14b37908eaf7d5506302987228bff69e1b8e7071ccd4e70fd0283b1b47f0b",
"sha256:e834249c45aa9837d0753351cdca61a4b8b383cc9ad0ff2325c97ff7b69e72a6",
"sha256:eed1b234c4499811ee85bcefa22ef5e466e75d132502226ed29740d593316c1f"
],
"version": "==2.0.0a1"
},
"multidict": {
"hashes": [
"sha256:317f96bc0950d249e96d8d29ab556d01dd38888fbe68324f46fd834b430169f1",
"sha256:42f56542166040b4474c0c608ed051732033cd821126493cf25b6c276df7dd35",
"sha256:4b7df040fb5fe826d689204f9b544af469593fb3ff3a069a6ad3409f742f5928",
"sha256:544fae9261232a97102e27a926019100a9db75bec7b37feedd74b3aa82f29969",
"sha256:620b37c3fea181dab09267cd5a84b0f23fa043beb8bc50d8474dd9694de1fa6e",
"sha256:6e6fef114741c4d7ca46da8449038ec8b1e880bbe68674c01ceeb1ac8a648e78",
"sha256:7774e9f6c9af3f12f296131453f7b81dabb7ebdb948483362f5afcaac8a826f1",
"sha256:85cb26c38c96f76b7ff38b86c9d560dea10cf3459bb5f4caf72fc1bb932c7136",
"sha256:a326f4240123a2ac66bb163eeba99578e9d63a8654a59f4688a79198f9aa10f8",
"sha256:ae402f43604e3b2bc41e8ea8b8526c7fa7139ed76b0d64fc48e28125925275b2",
"sha256:aee283c49601fa4c13adc64c09c978838a7e812f85377ae130a24d7198c0331e",
"sha256:b51249fdd2923739cd3efc95a3d6c363b67bbf779208e9f37fd5e68540d1a4d4",
"sha256:bb519becc46275c594410c6c28a8a0adc66fe24fef154a9addea54c1adb006f5",
"sha256:c2c37185fb0af79d5c117b8d2764f4321eeb12ba8c141a95d0aa8c2c1d0a11dd",
"sha256:dc561313279f9d05a3d0ffa89cd15ae477528ea37aa9795c4654588a3287a9ab",
"sha256:e439c9a10a95cb32abd708bb8be83b2134fa93790a4fb0535ca36db3dda94d20",
"sha256:fc3b4adc2ee8474cb3cd2a155305d5f8eda0a9c91320f83e55748e1fcb68f8e3"
],
"version": "==4.7.5"
},
"pathspec": {
"hashes": [
"sha256:7d91249d21749788d07a2d0f94147accd8f845507400749ea19c1ec9054a12b0",
"sha256:da45173eb3a6f2a5a487efba21f050af2b41948be6ab52b6a1e3ff22bb8b7061"
],
"version": "==0.8.0"
},
"protobuf": {
"hashes": [
"sha256:0bae429443cc4748be2aadfdaf9633297cfaeb24a9a02d0ab15849175ce90fab",
"sha256:24e3b6ad259544d717902777b33966a1a069208c885576254c112663e6a5bb0f",
"sha256:310a7aca6e7f257510d0c750364774034272538d51796ca31d42c3925d12a52a",
"sha256:52e586072612c1eec18e1174f8e3bb19d08f075fc2e3f91d3b16c919078469d0",
"sha256:73152776dc75f335c476d11d52ec6f0f6925774802cd48d6189f4d5d7fe753f4",
"sha256:7774bbbaac81d3ba86de646c39f154afc8156717972bf0450c9dbfa1dc8dbea2",
"sha256:82d7ac987715d8d1eb4068bf997f3053468e0ce0287e2729c30601feb6602fee",
"sha256:8eb9c93798b904f141d9de36a0ba9f9b73cc382869e67c9e642c0aba53b0fc07",
"sha256:adf0e4d57b33881d0c63bb11e7f9038f98ee0c3e334c221f0858f826e8fb0151",
"sha256:c40973a0aee65422d8cb4e7d7cbded95dfeee0199caab54d5ab25b63bce8135a",
"sha256:c77c974d1dadf246d789f6dad1c24426137c9091e930dbf50e0a29c1fcf00b1f",
"sha256:dd9aa4401c36785ea1b6fff0552c674bdd1b641319cb07ed1fe2392388e9b0d7",
"sha256:e11df1ac6905e81b815ab6fd518e79be0a58b5dc427a2cf7208980f30694b956",
"sha256:e2f8a75261c26b2f5f3442b0525d50fd79a71aeca04b5ec270fc123536188306",
"sha256:e512b7f3a4dd780f59f1bf22c302740e27b10b5c97e858a6061772668cd6f961",
"sha256:ef2c2e56aaf9ee914d3dccc3408d42661aaf7d9bb78eaa8f17b2e6282f214481",
"sha256:fac513a9dc2a74b99abd2e17109b53945e364649ca03d9f7a0b96aa8d1807d0a",
"sha256:fdfb6ad138dbbf92b5dbea3576d7c8ba7463173f7d2cb0ca1bd336ec88ddbd80"
],
"index": "pypi",
"version": "==3.11.3"
},
"regex": {
"hashes": [
"sha256:08119f707f0ebf2da60d2f24c2f39ca616277bb67ef6c92b72cbf90cbe3a556b",
"sha256:0ce9537396d8f556bcfc317c65b6a0705320701e5ce511f05fc04421ba05b8a8",
"sha256:1cbe0fa0b7f673400eb29e9ef41d4f53638f65f9a2143854de6b1ce2899185c3",
"sha256:2294f8b70e058a2553cd009df003a20802ef75b3c629506be20687df0908177e",
"sha256:23069d9c07e115537f37270d1d5faea3e0bdded8279081c4d4d607a2ad393683",
"sha256:24f4f4062eb16c5bbfff6a22312e8eab92c2c99c51a02e39b4eae54ce8255cd1",
"sha256:295badf61a51add2d428a46b8580309c520d8b26e769868b922750cf3ce67142",
"sha256:2a3bf8b48f8e37c3a40bb3f854bf0121c194e69a650b209628d951190b862de3",
"sha256:4385f12aa289d79419fede43f979e372f527892ac44a541b5446617e4406c468",
"sha256:5635cd1ed0a12b4c42cce18a8d2fb53ff13ff537f09de5fd791e97de27b6400e",
"sha256:5bfed051dbff32fd8945eccca70f5e22b55e4148d2a8a45141a3b053d6455ae3",
"sha256:7e1037073b1b7053ee74c3c6c0ada80f3501ec29d5f46e42669378eae6d4405a",
"sha256:90742c6ff121a9c5b261b9b215cb476eea97df98ea82037ec8ac95d1be7a034f",
"sha256:a58dd45cb865be0ce1d5ecc4cfc85cd8c6867bea66733623e54bd95131f473b6",
"sha256:c087bff162158536387c53647411db09b6ee3f9603c334c90943e97b1052a156",
"sha256:c162a21e0da33eb3d31a3ac17a51db5e634fc347f650d271f0305d96601dc15b",
"sha256:c9423a150d3a4fc0f3f2aae897a59919acd293f4cb397429b120a5fcd96ea3db",
"sha256:ccccdd84912875e34c5ad2d06e1989d890d43af6c2242c6fcfa51556997af6cd",
"sha256:e91ba11da11cf770f389e47c3f5c30473e6d85e06d7fd9dcba0017d2867aab4a",
"sha256:ea4adf02d23b437684cd388d557bf76e3afa72f7fed5bbc013482cc00c816948",
"sha256:fb95debbd1a824b2c4376932f2216cc186912e389bdb0e27147778cf6acb3f89"
],
"version": "==2020.4.4"
},
"six": {
"hashes": [
"sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a",
"sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c"
],
"version": "==1.14.0"
},
"stringcase": {
"hashes": [
"sha256:48a06980661908efe8d9d34eab2b6c13aefa2163b3ced26972902e3bdfd87008"
],
"index": "pypi",
"version": "==1.2.0"
},
"toml": {
"hashes": [
"sha256:229f81c57791a41d65e399fc06bf0848bab550a9dfd5ed66df18ce5f05e73d5c",
"sha256:235682dd292d5899d361a811df37e04a8828a5b1da3115886b73cf81ebc9100e"
],
"version": "==0.10.0"
},
"typed-ast": {
"hashes": [
"sha256:0666aa36131496aed8f7be0410ff974562ab7eeac11ef351def9ea6fa28f6355",
"sha256:0c2c07682d61a629b68433afb159376e24e5b2fd4641d35424e462169c0a7919",
"sha256:249862707802d40f7f29f6e1aad8d84b5aa9e44552d2cc17384b209f091276aa",
"sha256:24995c843eb0ad11a4527b026b4dde3da70e1f2d8806c99b7b4a7cf491612652",
"sha256:269151951236b0f9a6f04015a9004084a5ab0d5f19b57de779f908621e7d8b75",
"sha256:4083861b0aa07990b619bd7ddc365eb7fa4b817e99cf5f8d9cf21a42780f6e01",
"sha256:498b0f36cc7054c1fead3d7fc59d2150f4d5c6c56ba7fb150c013fbc683a8d2d",
"sha256:4e3e5da80ccbebfff202a67bf900d081906c358ccc3d5e3c8aea42fdfdfd51c1",
"sha256:6daac9731f172c2a22ade6ed0c00197ee7cc1221aa84cfdf9c31defeb059a907",
"sha256:715ff2f2df46121071622063fc7543d9b1fd19ebfc4f5c8895af64a77a8c852c",
"sha256:73d785a950fc82dd2a25897d525d003f6378d1cb23ab305578394694202a58c3",
"sha256:8c8aaad94455178e3187ab22c8b01a3837f8ee50e09cf31f1ba129eb293ec30b",
"sha256:8ce678dbaf790dbdb3eba24056d5364fb45944f33553dd5869b7580cdbb83614",
"sha256:aaee9905aee35ba5905cfb3c62f3e83b3bec7b39413f0a7f19be4e547ea01ebb",
"sha256:bcd3b13b56ea479b3650b82cabd6b5343a625b0ced5429e4ccad28a8973f301b",
"sha256:c9e348e02e4d2b4a8b2eedb48210430658df6951fa484e59de33ff773fbd4b41",
"sha256:d205b1b46085271b4e15f670058ce182bd1199e56b317bf2ec004b6a44f911f6",
"sha256:d43943ef777f9a1c42bf4e552ba23ac77a6351de620aa9acf64ad54933ad4d34",
"sha256:d5d33e9e7af3b34a40dc05f498939f0ebf187f07c385fd58d591c533ad8562fe",
"sha256:fc0fea399acb12edbf8a628ba8d2312f583bdbdb3335635db062fa98cf71fca4",
"sha256:fe460b922ec15dd205595c9b5b99e2f056fd98ae8f9f56b888e7a17dc2b757e7"
],
"version": "==1.4.1"
}
},
"develop": {
"attrs": {
"hashes": [
"sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c",
"sha256:f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72"
],
"version": "==19.3.0"
},
"entrypoints": {
"hashes": [
"sha256:589f874b313739ad35be6e0cd7efde2a4e9b6fea91edcc34e58ecbb8dbe56d19",
"sha256:c70dd71abe5a8c85e55e12c19bd91ccfeec11a6e99044204511f9ed547d48451"
],
"version": "==0.3"
},
"flake8": {
"hashes": [
"sha256:45681a117ecc81e870cbf1262835ae4af5e7a8b08e40b944a8a6e6b895914cfb",
"sha256:49356e766643ad15072a789a20915d3c91dc89fd313ccd71802303fd67e4deca"
],
"index": "pypi",
"version": "==3.7.9"
},
"importlib-metadata": {
"hashes": [
"sha256:2a688cbaa90e0cc587f1df48bdc97a6eadccdcd9c35fb3f976a09e3b5016d90f",
"sha256:34513a8a0c4962bc66d35b359558fd8a5e10cd472d37aec5f66858addef32c1e"
],
"markers": "python_version < '3.8'",
"version": "==1.6.0"
},
"isort": {
"hashes": [
"sha256:54da7e92468955c4fceacd0c86bd0ec997b0e1ee80d97f67c35a78b719dccab1",
"sha256:6e811fcb295968434526407adb8796944f1988c5b65e8139058f2014cbe100fd"
],
"index": "pypi",
"version": "==4.3.21"
},
"mccabe": {
"hashes": [
"sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
"sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"
],
"version": "==0.6.1"
},
"more-itertools": {
"hashes": [
"sha256:5dd8bcf33e5f9513ffa06d5ad33d78f31e1931ac9a18f33d37e77a180d393a7c",
"sha256:b1ddb932186d8a6ac451e1d95844b382f55e12686d51ca0c68b6f61f2ab7a507"
],
"version": "==8.2.0"
},
"mypy": {
"hashes": [
"sha256:15b948e1302682e3682f11f50208b726a246ab4e6c1b39f9264a8796bb416aa2",
"sha256:219a3116ecd015f8dca7b5d2c366c973509dfb9a8fc97ef044a36e3da66144a1",
"sha256:3b1fc683fb204c6b4403a1ef23f0b1fac8e4477091585e0c8c54cbdf7d7bb164",
"sha256:3beff56b453b6ef94ecb2996bea101a08f1f8a9771d3cbf4988a61e4d9973761",
"sha256:7687f6455ec3ed7649d1ae574136835a4272b65b3ddcf01ab8704ac65616c5ce",
"sha256:7ec45a70d40ede1ec7ad7f95b3c94c9cf4c186a32f6bacb1795b60abd2f9ef27",
"sha256:86c857510a9b7c3104cf4cde1568f4921762c8f9842e987bc03ed4f160925754",
"sha256:8a627507ef9b307b46a1fea9513d5c98680ba09591253082b4c48697ba05a4ae",
"sha256:8dfb69fbf9f3aeed18afffb15e319ca7f8da9642336348ddd6cab2713ddcf8f9",
"sha256:a34b577cdf6313bf24755f7a0e3f3c326d5c1f4fe7422d1d06498eb25ad0c600",
"sha256:a8ffcd53cb5dfc131850851cc09f1c44689c2812d0beb954d8138d4f5fc17f65",
"sha256:b90928f2d9eb2f33162405f32dde9f6dcead63a0971ca8a1b50eb4ca3e35ceb8",
"sha256:c56ffe22faa2e51054c5f7a3bc70a370939c2ed4de308c690e7949230c995913",
"sha256:f91c7ae919bbc3f96cd5e5b2e786b2b108343d1d7972ea130f7de27fdd547cf3"
],
"index": "pypi",
"version": "==0.770"
},
"mypy-extensions": {
"hashes": [
"sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d",
"sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"
],
"version": "==0.4.3"
},
"packaging": {
"hashes": [
"sha256:3c292b474fda1671ec57d46d739d072bfd495a4f51ad01a055121d81e952b7a3",
"sha256:82f77b9bee21c1bafbf35a84905d604d5d1223801d639cf3ed140bd651c08752"
],
"version": "==20.3"
},
"pluggy": {
"hashes": [
"sha256:15b2acde666561e1298d71b523007ed7364de07029219b604cf808bfa1c765b0",
"sha256:966c145cd83c96502c3c3868f50408687b38434af77734af1e9ca461a4081d2d"
],
"version": "==0.13.1"
},
"py": {
"hashes": [
"sha256:5e27081401262157467ad6e7f851b7aa402c5852dbcb3dae06768434de5752aa",
"sha256:c20fdd83a5dbc0af9efd622bee9a5564e278f6380fffcacc43ba6f43db2813b0"
],
"version": "==1.8.1"
},
"pycodestyle": {
"hashes": [
"sha256:95a2219d12372f05704562a14ec30bc76b05a5b297b21a5dfe3f6fac3491ae56",
"sha256:e40a936c9a450ad81df37f549d676d127b1b66000a6c500caa2b085bc0ca976c"
],
"version": "==2.5.0"
},
"pyflakes": {
"hashes": [
"sha256:17dbeb2e3f4d772725c777fabc446d5634d1038f234e77343108ce445ea69ce0",
"sha256:d976835886f8c5b31d47970ed689944a0262b5f3afa00a5a7b4dc81e5449f8a2"
],
"version": "==2.1.1"
},
"pyparsing": {
"hashes": [
"sha256:67199f0c41a9c702154efb0e7a8cc08accf830eb003b4d9fa42c4059002e2492",
"sha256:700d17888d441604b0bd51535908dcb297561b040819cccde647a92439db5a2a"
],
"version": "==3.0.0a1"
},
"pytest": {
"hashes": [
"sha256:0e5b30f5cb04e887b91b1ee519fa3d89049595f428c1db76e73bd7f17b09b172",
"sha256:84dde37075b8805f3d1f392cc47e38a0e59518fb46a431cfdaf7cf1ce805f970"
],
"index": "pypi",
"version": "==5.4.1"
},
"pytest-asyncio": {
"hashes": [
"sha256:9fac5100fd716cbecf6ef89233e8590a4ad61d729d1732e0a96b84182df1daaf",
"sha256:d734718e25cfc32d2bf78d346e99d33724deeba774cc4afdf491530c6184b63b"
],
"index": "pypi",
"version": "==0.10.0"
},
"rope": {
"hashes": [
"sha256:52423a7eebb5306a6d63bdc91a7c657db51ac9babfb8341c9a1440831ecf3203",
"sha256:ae1fa2fd56f64f4cc9be46493ce54bed0dd12dee03980c61a4393d89d84029ad",
"sha256:d2830142c2e046f5fc26a022fe680675b6f48f81c7fc1f03a950706e746e9dfe"
],
"index": "pypi",
"version": "==0.16.0"
},
"six": {
"hashes": [
"sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a",
"sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c"
],
"version": "==1.14.0"
},
"typed-ast": {
"hashes": [
"sha256:0666aa36131496aed8f7be0410ff974562ab7eeac11ef351def9ea6fa28f6355",
"sha256:0c2c07682d61a629b68433afb159376e24e5b2fd4641d35424e462169c0a7919",
"sha256:249862707802d40f7f29f6e1aad8d84b5aa9e44552d2cc17384b209f091276aa",
"sha256:24995c843eb0ad11a4527b026b4dde3da70e1f2d8806c99b7b4a7cf491612652",
"sha256:269151951236b0f9a6f04015a9004084a5ab0d5f19b57de779f908621e7d8b75",
"sha256:4083861b0aa07990b619bd7ddc365eb7fa4b817e99cf5f8d9cf21a42780f6e01",
"sha256:498b0f36cc7054c1fead3d7fc59d2150f4d5c6c56ba7fb150c013fbc683a8d2d",
"sha256:4e3e5da80ccbebfff202a67bf900d081906c358ccc3d5e3c8aea42fdfdfd51c1",
"sha256:6daac9731f172c2a22ade6ed0c00197ee7cc1221aa84cfdf9c31defeb059a907",
"sha256:715ff2f2df46121071622063fc7543d9b1fd19ebfc4f5c8895af64a77a8c852c",
"sha256:73d785a950fc82dd2a25897d525d003f6378d1cb23ab305578394694202a58c3",
"sha256:8c8aaad94455178e3187ab22c8b01a3837f8ee50e09cf31f1ba129eb293ec30b",
"sha256:8ce678dbaf790dbdb3eba24056d5364fb45944f33553dd5869b7580cdbb83614",
"sha256:aaee9905aee35ba5905cfb3c62f3e83b3bec7b39413f0a7f19be4e547ea01ebb",
"sha256:bcd3b13b56ea479b3650b82cabd6b5343a625b0ced5429e4ccad28a8973f301b",
"sha256:c9e348e02e4d2b4a8b2eedb48210430658df6951fa484e59de33ff773fbd4b41",
"sha256:d205b1b46085271b4e15f670058ce182bd1199e56b317bf2ec004b6a44f911f6",
"sha256:d43943ef777f9a1c42bf4e552ba23ac77a6351de620aa9acf64ad54933ad4d34",
"sha256:d5d33e9e7af3b34a40dc05f498939f0ebf187f07c385fd58d591c533ad8562fe",
"sha256:fc0fea399acb12edbf8a628ba8d2312f583bdbdb3335635db062fa98cf71fca4",
"sha256:fe460b922ec15dd205595c9b5b99e2f056fd98ae8f9f56b888e7a17dc2b757e7"
],
"version": "==1.4.1"
},
"typing-extensions": {
"hashes": [
"sha256:6e95524d8a547a91e08f404ae485bbb71962de46967e1b71a0cb89af24e761c5",
"sha256:79ee589a3caca649a9bfd2a8de4709837400dfa00b6cc81962a1e6a1815969ae",
"sha256:f8d2bd89d25bc39dabe7d23df520442fa1d8969b82544370e03d88b5a591c392"
],
"version": "==3.7.4.2"
},
"v": {
"hashes": [
"sha256:2d5a8f79a36aaebe62ef2c7068e3ec7f86656078202edabfdbf74715dc822d36",
"sha256:cd6b6b20b4a611f209c88bcdfb7211321f85662efb2bdd53a7b40314d0a84618"
],
"index": "pypi",
"version": "==0.0.0"
},
"wcwidth": {
"hashes": [
"sha256:cafe2186b3c009a04067022ce1dcd79cb38d8d65ee4f4791b8888d6599d1bbe1",
"sha256:ee73862862a156bf77ff92b09034fc4825dd3af9cf81bc5b360668d425f3c5f1"
],
"version": "==0.1.9"
},
"zipp": {
"hashes": [
"sha256:aa36550ff0c0b7ef7fa639055d797116ee891440eac1a56f378e2d3179e0320b",
"sha256:c599e4d75c98f6798c509911d08a22e6c021d074469042177c8c86fb92eefd96"
],
"version": "==3.1.0"
}
}
}

View File

@@ -311,10 +311,26 @@ $ pip install -e .
There are two types of tests:
1. Manually-written tests for some behavior of the library
2. Proto files and JSON inputs for automated tests
1. Standard tests
2. Custom tests
For #2, you can add a new `*.proto` file into the `betterproto/tests` directory along with a sample `*.json` input and it will get automatically picked up.
#### Standard tests
Adding a standard test case is easy.
- Create a new directory `betterproto/tests/inputs/<name>`
- add `<name>.proto` with a message called `Test`
- add `<name>.json` with some test data
It will be picked up automatically when you run the tests.
- See also: [Standard Tests Development Guide](betterproto/tests/README.md)
#### Custom tests
Custom tests are found in `tests/test_*.py` and are run with pytest.
#### Running
Here's how to run the tests.
@@ -322,7 +338,7 @@ Here's how to run the tests.
# Generate assets from sample .proto files
$ pipenv run generate
# Run the tests
# Run all tests
$ pipenv run test
```
@@ -340,6 +356,9 @@ $ pipenv run test
- [x] Refs to nested types
- [x] Imports in proto files
- [x] Well-known Google types
- [ ] Support as request input
- [ ] Support as response output
- [ ] Automatically wrap/unwrap responses
- [x] OneOf support
- [x] Basic support on the wire
- [x] Check which was set from the group

View File

@@ -11,10 +11,12 @@ from typing import (
Any,
AsyncGenerator,
Callable,
Collection,
Dict,
Generator,
Iterable,
List,
Mapping,
Optional,
SupportsBytes,
Tuple,
@@ -25,7 +27,7 @@ from typing import (
TYPE_CHECKING,
)
import grpclib.client
import grpclib.const
import stringcase
@@ -33,6 +35,8 @@ from .casing import safe_snake_case
if TYPE_CHECKING:
from grpclib._protocols import IProtoMessage
from grpclib.client import Channel
from grpclib.metadata import Deadline
if not (sys.version_info.major == 3 and sys.version_info.minor >= 7):
# Apply backport of datetime.fromisoformat from 3.7
@@ -118,7 +122,11 @@ WIRE_LEN_DELIM_TYPES = [TYPE_STRING, TYPE_BYTES, TYPE_MESSAGE, TYPE_MAP]
# Protobuf datetimes start at the Unix Epoch in 1970 in UTC.
DATETIME_ZERO = datetime(1970, 1, 1, tzinfo=timezone.utc)
def datetime_default_gen():
return datetime(1970, 1, 1, tzinfo=timezone.utc)
DATETIME_ZERO = datetime_default_gen()
class Casing(enum.Enum):
@@ -426,6 +434,63 @@ def parse_fields(value: bytes) -> Generator[ParsedField, None, None]:
T = TypeVar("T", bound="Message")
class ProtoClassMetadata:
cls: Type["Message"]
def __init__(self, cls: Type["Message"]):
self.cls = cls
by_field = {}
by_group = {}
for field in dataclasses.fields(cls):
meta = FieldMetadata.get(field)
if meta.group:
# This is part of a one-of group.
by_field[field.name] = meta.group
by_group.setdefault(meta.group, set()).add(field)
self.oneof_group_by_field = by_field
self.oneof_field_by_group = by_group
self.init_default_gen()
self.init_cls_by_field()
def init_default_gen(self):
default_gen = {}
for field in dataclasses.fields(self.cls):
meta = FieldMetadata.get(field)
default_gen[field.name] = self.cls._get_field_default_gen(field, meta)
self.default_gen = default_gen
def init_cls_by_field(self):
field_cls = {}
for field in dataclasses.fields(self.cls):
meta = FieldMetadata.get(field)
if meta.proto_type == TYPE_MAP:
assert meta.map_types
kt = self.cls._cls_for(field, index=0)
vt = self.cls._cls_for(field, index=1)
Entry = dataclasses.make_dataclass(
"Entry",
[
("key", kt, dataclass_field(1, meta.map_types[0])),
("value", vt, dataclass_field(2, meta.map_types[1])),
],
bases=(Message,),
)
field_cls[field.name] = Entry
field_cls[field.name + ".value"] = vt
else:
field_cls[field.name] = self.cls._cls_for(field)
self.cls_by_field = field_cls
class Message(ABC):
"""
A protobuf message base class. Generated code will inherit from this and
@@ -443,17 +508,12 @@ class Message(ABC):
# Set a default value for each field in the class after `__init__` has
# already been run.
group_map: Dict[str, dict] = {"fields": {}, "groups": {}}
group_map: Dict[str, dataclasses.Field] = {}
for field in dataclasses.fields(self):
meta = FieldMetadata.get(field)
if meta.group:
# This is part of a one-of group.
group_map["fields"][field.name] = meta.group
if meta.group not in group_map["groups"]:
group_map["groups"][meta.group] = {"current": None, "fields": set()}
group_map["groups"][meta.group]["fields"].add(field)
group_map.setdefault(meta.group)
if getattr(self, field.name) != PLACEHOLDER:
# Skip anything not set to the sentinel value
@@ -461,7 +521,7 @@ class Message(ABC):
if meta.group:
# This was set, so make it the selected value of the one-of.
group_map["groups"][meta.group]["current"] = field
group_map[meta.group] = field
continue
@@ -477,19 +537,33 @@ class Message(ABC):
# Track when a field has been set.
self.__dict__["_serialized_on_wire"] = True
if attr in getattr(self, "_group_map", {}).get("fields", {}):
group = self._group_map["fields"][attr]
for field in self._group_map["groups"][group]["fields"]:
if field.name == attr:
self._group_map["groups"][group]["current"] = field
else:
super().__setattr__(
field.name,
self._get_field_default(field, FieldMetadata.get(field)),
)
if hasattr(self, "_group_map"): # __post_init__ had already run
if attr in self._betterproto.oneof_group_by_field:
group = self._betterproto.oneof_group_by_field[attr]
for field in self._betterproto.oneof_field_by_group[group]:
if field.name == attr:
self._group_map[group] = field
else:
super().__setattr__(
field.name,
self._get_field_default(field, FieldMetadata.get(field)),
)
super().__setattr__(attr, value)
@property
def _betterproto(self):
"""
Lazy initialize metadata for each protobuf class.
It may be initialized multiple times in a multi-threaded environment,
but that won't affect the correctness.
"""
meta = getattr(self.__class__, "_betterproto_meta", None)
if not meta:
meta = ProtoClassMetadata(self.__class__)
self.__class__._betterproto_meta = meta
return meta
def __bytes__(self) -> bytes:
"""
Get the binary encoded Protobuf representation of this instance.
@@ -508,7 +582,7 @@ class Message(ABC):
# currently set in a `oneof` group, so it must be serialized even
# if the value is the default zero value.
selected_in_group = False
if meta.group and self._group_map["groups"][meta.group]["current"] == field:
if meta.group and self._group_map[meta.group] == field:
selected_in_group = True
serialize_empty = False
@@ -560,47 +634,50 @@ class Message(ABC):
# For compatibility with other libraries
SerializeToString = __bytes__
def _type_hint(self, field_name: str) -> Type:
module = inspect.getmodule(self.__class__)
type_hints = get_type_hints(self.__class__, vars(module))
@classmethod
def _type_hint(cls, field_name: str) -> Type:
module = inspect.getmodule(cls)
type_hints = get_type_hints(cls, vars(module))
return type_hints[field_name]
def _cls_for(self, field: dataclasses.Field, index: int = 0) -> Type:
@classmethod
def _cls_for(cls, field: dataclasses.Field, index: int = 0) -> Type:
"""Get the message class for a field from the type hints."""
cls = self._type_hint(field.name)
if hasattr(cls, "__args__") and index >= 0:
cls = cls.__args__[index]
return cls
field_cls = cls._type_hint(field.name)
if hasattr(field_cls, "__args__") and index >= 0:
field_cls = field_cls.__args__[index]
return field_cls
def _get_field_default(self, field: dataclasses.Field, meta: FieldMetadata) -> Any:
t = self._type_hint(field.name)
return self._betterproto.default_gen[field.name]()
@classmethod
def _get_field_default_gen(cls, field: dataclasses.Field, meta: FieldMetadata) -> Any:
t = cls._type_hint(field.name)
value: Any = 0
if hasattr(t, "__origin__"):
if t.__origin__ in (dict, Dict):
# This is some kind of map (dict in Python).
value = {}
return dict
elif t.__origin__ in (list, List):
# This is some kind of list (repeated) field.
value = []
return list
elif t.__origin__ == Union and t.__args__[1] == type(None):
# This is an optional (wrapped) field. For setting the default we
# really don't care what kind of field it is.
value = None
return type(None)
else:
value = t()
return t
elif issubclass(t, Enum):
# Enums always default to zero.
value = 0
return int
elif t == datetime:
# Offsets are relative to 1970-01-01T00:00:00Z
value = DATETIME_ZERO
return datetime_default_gen
else:
# This is either a primitive scalar or another message type. Calling
# it should result in its zero value.
value = t()
return value
return t
def _postprocess_single(
self, wire_type: int, meta: FieldMetadata, field: dataclasses.Field, value: Any
@@ -625,7 +702,7 @@ class Message(ABC):
if meta.proto_type == TYPE_STRING:
value = value.decode("utf-8")
elif meta.proto_type == TYPE_MESSAGE:
cls = self._cls_for(field)
cls = self._betterproto.cls_by_field[field.name]
if cls == datetime:
value = _Timestamp().parse(value).to_datetime()
@@ -639,20 +716,7 @@ class Message(ABC):
value = cls().parse(value)
value._serialized_on_wire = True
elif meta.proto_type == TYPE_MAP:
# TODO: This is slow, use a cache to make it faster since each
# key/value pair will recreate the class.
assert meta.map_types
kt = self._cls_for(field, index=0)
vt = self._cls_for(field, index=1)
Entry = dataclasses.make_dataclass(
"Entry",
[
("key", kt, dataclass_field(1, meta.map_types[0])),
("value", vt, dataclass_field(2, meta.map_types[1])),
],
bases=(Message,),
)
value = Entry().parse(value)
value = self._betterproto.cls_by_field[field.name]().parse(value)
return value
@@ -767,7 +831,7 @@ class Message(ABC):
else:
output[cased_name] = b64encode(v).decode("utf8")
elif meta.proto_type == TYPE_ENUM:
enum_values = list(self._cls_for(field)) # type: ignore
enum_values = list(self._betterproto.cls_by_field[field.name]) # type: ignore
if isinstance(v, list):
output[cased_name] = [enum_values[e].name for e in v]
else:
@@ -793,7 +857,7 @@ class Message(ABC):
if meta.proto_type == "message":
v = getattr(self, field.name)
if isinstance(v, list):
cls = self._cls_for(field)
cls = self._betterproto.cls_by_field[field.name]
for i in range(len(value[key])):
v.append(cls().from_dict(value[key][i]))
elif isinstance(v, datetime):
@@ -810,7 +874,7 @@ class Message(ABC):
v.from_dict(value[key])
elif meta.map_types and meta.map_types[1] == TYPE_MESSAGE:
v = getattr(self, field.name)
cls = self._cls_for(field, index=1)
cls = self._betterproto.cls_by_field[field.name + ".value"]
for k in value[key]:
v[k] = cls().from_dict(value[key][k])
else:
@@ -826,7 +890,7 @@ class Message(ABC):
else:
v = b64decode(value[key])
elif meta.proto_type == TYPE_ENUM:
enum_cls = self._cls_for(field)
enum_cls = self._betterproto.cls_by_field[field.name]
if isinstance(v, list):
v = [enum_cls.from_string(e) for e in v]
elif isinstance(v, str):
@@ -859,7 +923,7 @@ def serialized_on_wire(message: Message) -> bool:
def which_one_of(message: Message, group_name: str) -> Tuple[str, Any]:
"""Return the name and value of a message's one-of field group."""
field = message._group_map["groups"].get(group_name, {}).get("current")
field = message._group_map.get(group_name)
if not field:
return ("", None)
return (field.name, getattr(message, field.name))
@@ -1000,20 +1064,57 @@ def _get_wrapper(proto_type: str) -> Type:
}[proto_type]
_Value = Union[str, bytes]
_MetadataLike = Union[Mapping[str, _Value], Collection[Tuple[str, _Value]]]
class ServiceStub(ABC):
"""
Base class for async gRPC service stubs.
"""
def __init__(self, channel: grpclib.client.Channel) -> None:
def __init__(
self,
channel: "Channel",
*,
timeout: Optional[float] = None,
deadline: Optional["Deadline"] = None,
metadata: Optional[_MetadataLike] = None,
) -> None:
self.channel = channel
self.timeout = timeout
self.deadline = deadline
self.metadata = metadata
def __resolve_request_kwargs(
self,
timeout: Optional[float],
deadline: Optional["Deadline"],
metadata: Optional[_MetadataLike],
):
return {
"timeout": self.timeout if timeout is None else timeout,
"deadline": self.deadline if deadline is None else deadline,
"metadata": self.metadata if metadata is None else metadata,
}
async def _unary_unary(
self, route: str, request: "IProtoMessage", response_type: Type[T]
self,
route: str,
request: "IProtoMessage",
response_type: Type[T],
*,
timeout: Optional[float] = None,
deadline: Optional["Deadline"] = None,
metadata: Optional[_MetadataLike] = None,
) -> T:
"""Make a unary request and return the response."""
async with self.channel.request(
route, grpclib.const.Cardinality.UNARY_UNARY, type(request), response_type
route,
grpclib.const.Cardinality.UNARY_UNARY,
type(request),
response_type,
**self.__resolve_request_kwargs(timeout, deadline, metadata),
) as stream:
await stream.send_message(request, end=True)
response = await stream.recv_message()
@@ -1021,11 +1122,22 @@ class ServiceStub(ABC):
return response
async def _unary_stream(
self, route: str, request: "IProtoMessage", response_type: Type[T]
self,
route: str,
request: "IProtoMessage",
response_type: Type[T],
*,
timeout: Optional[float] = None,
deadline: Optional["Deadline"] = None,
metadata: Optional[_MetadataLike] = None,
) -> AsyncGenerator[T, None]:
"""Make a unary request and return the stream response iterator."""
async with self.channel.request(
route, grpclib.const.Cardinality.UNARY_STREAM, type(request), response_type
route,
grpclib.const.Cardinality.UNARY_STREAM,
type(request),
response_type,
**self.__resolve_request_kwargs(timeout, deadline, metadata),
) as stream:
await stream.send_message(request, end=True)
async for message in stream:

2
betterproto/plugin.bat Normal file
View File

@@ -0,0 +1,2 @@
@SET plugin_dir=%~dp0
@python %plugin_dir%/plugin.py %*

View File

@@ -1,12 +1,11 @@
#!/usr/bin/env python
import itertools
import json
import os.path
import re
import sys
import textwrap
from typing import Any, List, Tuple
from collections import defaultdict
from typing import Dict, List, Optional, Type
try:
import black
@@ -24,44 +23,51 @@ from google.protobuf.descriptor_pb2 import (
DescriptorProto,
EnumDescriptorProto,
FieldDescriptorProto,
FileDescriptorProto,
ServiceDescriptorProto,
)
from betterproto.casing import safe_snake_case
import google.protobuf.wrappers_pb2 as google_wrappers
WRAPPER_TYPES = {
"google.protobuf.DoubleValue": "float",
"google.protobuf.FloatValue": "float",
"google.protobuf.Int64Value": "int",
"google.protobuf.UInt64Value": "int",
"google.protobuf.Int32Value": "int",
"google.protobuf.UInt32Value": "int",
"google.protobuf.BoolValue": "bool",
"google.protobuf.StringValue": "str",
"google.protobuf.BytesValue": "bytes",
}
WRAPPER_TYPES: Dict[str, Optional[Type]] = defaultdict(lambda: None, {
'google.protobuf.DoubleValue': google_wrappers.DoubleValue,
'google.protobuf.FloatValue': google_wrappers.FloatValue,
'google.protobuf.Int64Value': google_wrappers.Int64Value,
'google.protobuf.UInt64Value': google_wrappers.UInt64Value,
'google.protobuf.Int32Value': google_wrappers.Int32Value,
'google.protobuf.UInt32Value': google_wrappers.UInt32Value,
'google.protobuf.BoolValue': google_wrappers.BoolValue,
'google.protobuf.StringValue': google_wrappers.StringValue,
'google.protobuf.BytesValue': google_wrappers.BytesValue,
})
def get_ref_type(package: str, imports: set, type_name: str) -> str:
def get_ref_type(package: str, imports: set, type_name: str, unwrap: bool = True) -> str:
"""
Return a Python type name for a proto type reference. Adds the import if
necessary.
necessary. Unwraps well known type if required.
"""
# If the package name is a blank string, then this should still work
# because by convention packages are lowercase and message/enum types are
# pascal-cased. May require refactoring in the future.
type_name = type_name.lstrip(".")
if type_name in WRAPPER_TYPES:
return f"Optional[{WRAPPER_TYPES[type_name]}]"
# Check if type is wrapper.
wrapper_class = WRAPPER_TYPES[type_name]
if type_name == "google.protobuf.Duration":
return "timedelta"
if unwrap:
if wrapper_class:
wrapped_type = type(wrapper_class().value)
return f"Optional[{wrapped_type.__name__}]"
if type_name == "google.protobuf.Timestamp":
return "datetime"
if type_name == "google.protobuf.Duration":
return "timedelta"
if type_name == "google.protobuf.Timestamp":
return "datetime"
elif wrapper_class:
imports.add(f"from {wrapper_class.__module__} import {wrapper_class.__name__}")
return f"{wrapper_class.__name__}"
if type_name.startswith(package):
parts = type_name.lstrip(package).lstrip(".").split(".")
@@ -122,7 +128,7 @@ def get_py_zero(type_num: int) -> str:
def traverse(proto_file):
def _traverse(path, items, prefix = ''):
def _traverse(path, items, prefix=""):
for i, item in enumerate(items):
# Adjust the name since we flatten the heirarchy.
item.name = next_prefix = prefix + item.name
@@ -379,7 +385,7 @@ def generate_code(request, response):
).strip('"'),
"input_message": input_message,
"output": get_ref_type(
package, output["imports"], method.output_type
package, output["imports"], method.output_type, unwrap=False
).strip('"'),
"client_streaming": method.client_streaming,
"server_streaming": method.server_streaming,

View File

@@ -63,7 +63,7 @@ class {{ service.py_name }}Stub(betterproto.ServiceStub):
{% endif %}
{% for method in service.methods %}
async def {{ method.py_name }}(self{% if method.input_message and method.input_message.properties %}, *, {% for field in method.input_message.properties %}{{ field.name }}: {% if field.zero == "None" and not field.type.startswith("Optional[") %}Optional[{{ field.type }}]{% else %}{{ field.type }}{% endif %} = {{ field.zero }}{% if not loop.last %}, {% endif %}{% endfor %}{% endif %}) -> {% if method.server_streaming %}AsyncGenerator[{{ method.output }}, None]{% else %}{{ method.output }}{% endif %}:
async def {{ method.py_name }}(self{% if method.input_message and method.input_message.properties %}, *, {% for field in method.input_message.properties %}{{ field.py_name }}: {% if field.zero == "None" and not field.type.startswith("Optional[") %}Optional[{{ field.type }}]{% else %}{{ field.type }}{% endif %} = {{ field.zero }}{% if not loop.last %}, {% endif %}{% endfor %}{% endif %}) -> {% if method.server_streaming %}AsyncGenerator[{{ method.output }}, None]{% else %}{{ method.output }}{% endif %}:
{% if method.comment %}
{{ method.comment }}
@@ -71,10 +71,10 @@ class {{ service.py_name }}Stub(betterproto.ServiceStub):
request = {{ method.input }}()
{% for field in method.input_message.properties %}
{% if field.field_type == 'message' %}
if {{ field.name }} is not None:
request.{{ field.name }} = {{ field.name }}
if {{ field.py_name }} is not None:
request.{{ field.py_name }} = {{ field.py_name }}
{% else %}
request.{{ field.name }} = {{ field.name }}
request.{{ field.py_name }} = {{ field.py_name }}
{% endif %}
{% endfor %}

View File

@@ -0,0 +1,90 @@
# Standard Tests Development Guide
Standard test cases are found in [betterproto/tests/inputs](inputs), where each subdirectory represents a testcase, that is verified in isolation.
```
inputs/
bool/
double/
int32/
...
```
## Test case directory structure
Each testcase has a `<name>.proto` file with a message called `Test`, a matching `.json` file and optionally a custom test file called `test_*.py`.
```bash
bool/
bool.proto
bool.json
test_bool.py # optional
```
### proto
`<name>.proto` &mdash; *The protobuf message to test*
```protobuf
syntax = "proto3";
message Test {
bool value = 1;
}
```
You can add multiple `.proto` files to the test case, as long as one file matches the directory name.
### json
`<name>.json` &mdash; *Test-data to validate the message with*
```json
{
"value": true
}
```
### pytest
`test_<name>.py` &mdash; *Custom test to validate specific aspects of the generated class*
```python
from betterproto.tests.output_betterproto.bool.bool import Test
def test_value():
message = Test()
assert not message.value, "Boolean is False by default"
```
## Standard tests
The following tests are automatically executed for all cases:
- [x] Can the generated python code imported?
- [x] Can the generated message class be instantiated?
- [x] Is the generated code compatible with the Google's `grpc_tools.protoc` implementation?
## Running the tests
- `pipenv run generate`
This generates
- `betterproto/tests/output_betterproto` &mdash; *the plugin generated python classes*
- `betterproto/tests/output_reference` &mdash; *reference implementation classes*
- `pipenv run test`
## Intentionally Failing tests
The standard test suite includes tests that fail by intention. These tests document known bugs and missing features that are intended to be corrented in the future.
When running `pytest`, they show up as `x` or `X` in the test results.
```
betterproto/tests/test_inputs.py ..x...x..x...x.X........xx........x.....x.......x.xx....x...................... [ 84%]
```
- `.` &mdash; PASSED
- `x` &mdash; XFAIL: expected failure
- `X` &mdash; XPASS: expected failure, but still passed
Test cases marked for expected failure are declared in [inputs/xfail.py](inputs.xfail.py)

View File

@@ -1,84 +1,91 @@
#!/usr/bin/env python
import glob
import os
import shutil
import sys
from typing import Set
from betterproto.tests.util import (
get_directories,
inputs_path,
output_path_betterproto,
output_path_reference,
protoc_plugin,
protoc_reference,
)
# Force pure-python implementation instead of C++, otherwise imports
# break things because we can't properly reset the symbol database.
os.environ["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"] = "python"
import importlib
import json
import subprocess
import sys
from typing import Generator, Tuple
from google.protobuf import symbol_database
from google.protobuf.descriptor_pool import DescriptorPool
from google.protobuf.json_format import MessageToJson, Parse
def clear_directory(path: str):
for file_or_directory in glob.glob(os.path.join(path, "*")):
if os.path.isdir(file_or_directory):
shutil.rmtree(file_or_directory)
else:
os.remove(file_or_directory)
root = os.path.dirname(os.path.realpath(__file__))
def generate(whitelist: Set[str]):
path_whitelist = {os.path.realpath(e) for e in whitelist if os.path.exists(e)}
name_whitelist = {e for e in whitelist if not os.path.exists(e)}
test_case_names = set(get_directories(inputs_path))
for test_case_name in sorted(test_case_names):
test_case_input_path = os.path.realpath(
os.path.join(inputs_path, test_case_name)
)
if (
whitelist
and test_case_input_path not in path_whitelist
and test_case_name not in name_whitelist
):
continue
test_case_output_path_reference = os.path.join(
output_path_reference, test_case_name
)
test_case_output_path_betterproto = os.path.join(
output_path_betterproto, test_case_name
)
print(f"Generating output for {test_case_name}")
os.makedirs(test_case_output_path_reference, exist_ok=True)
os.makedirs(test_case_output_path_betterproto, exist_ok=True)
clear_directory(test_case_output_path_reference)
clear_directory(test_case_output_path_betterproto)
protoc_reference(test_case_input_path, test_case_output_path_reference)
protoc_plugin(test_case_input_path, test_case_output_path_betterproto)
def get_files(end: str) -> Generator[str, None, None]:
for r, dirs, files in os.walk(root):
for filename in [f for f in files if f.endswith(end)]:
yield os.path.join(r, filename)
HELP = "\n".join(
[
"Usage: python generate.py",
" python generate.py [DIRECTORIES or NAMES]",
"Generate python classes for standard tests.",
"",
"DIRECTORIES One or more relative or absolute directories of test-cases to generate classes for.",
" python generate.py inputs/bool inputs/double inputs/enum",
"",
"NAMES One or more test-case names to generate classes for.",
" python generate.py bool double enums",
]
)
def get_base(filename: str) -> str:
return os.path.splitext(os.path.basename(filename))[0]
def main():
if set(sys.argv).intersection({"-h", "--help"}):
print(HELP)
return
whitelist = set(sys.argv[1:])
def ensure_ext(filename: str, ext: str) -> str:
if not filename.endswith(ext):
return filename + ext
return filename
generate(whitelist)
if __name__ == "__main__":
os.chdir(root)
if len(sys.argv) > 1:
proto_files = [ensure_ext(f, ".proto") for f in sys.argv[1:]]
bases = {get_base(f) for f in proto_files}
json_files = [
f for f in get_files(".json") if get_base(f).split("-")[0] in bases
]
else:
proto_files = get_files(".proto")
json_files = get_files(".json")
for filename in proto_files:
print(f"Generating code for {os.path.basename(filename)}")
subprocess.run(
f"protoc --python_out=. {os.path.basename(filename)}", shell=True
)
subprocess.run(
f"protoc --plugin=protoc-gen-custom=../plugin.py --custom_out=. {os.path.basename(filename)}",
shell=True,
)
for filename in json_files:
# Reset the internal symbol database so we can import the `Test` message
# multiple times. Ugh.
sym = symbol_database.Default()
sym.pool = DescriptorPool()
parts = get_base(filename).split("-")
out = filename.replace(".json", ".bin")
print(f"Using {parts[0]}_pb2 to generate {os.path.basename(out)}")
imported = importlib.import_module(f"{parts[0]}_pb2")
input_json = open(filename).read()
parsed = Parse(input_json, imported.Test())
serialized = parsed.SerializeToString()
preserve = "casing" not in filename
serialized_json = MessageToJson(parsed, preserving_proto_field_name=preserve)
s_loaded = json.loads(serialized_json)
in_loaded = json.loads(input_json)
if s_loaded != in_loaded:
raise AssertionError("Expected JSON to be equal:", s_loaded, in_loaded)
open(out, "wb").write(serialized)
main()

View File

@@ -0,0 +1,6 @@
from betterproto.tests.output_betterproto.bool.bool import Test
def test_value():
message = Test()
assert not message.value, "Boolean is False by default"

View File

@@ -9,4 +9,9 @@ enum my_enum {
message Test {
int32 camelCase = 1;
my_enum snake_case = 2;
snake_case_message snake_case_message = 3;
}
message snake_case_message {
}

View File

@@ -0,0 +1,22 @@
import betterproto.tests.output_betterproto.casing.casing as casing
from betterproto.tests.output_betterproto.casing.casing import Test
def test_message_attributes():
message = Test()
assert hasattr(
message, "snake_case_message"
), "snake_case field name is same in python"
assert hasattr(message, "camel_case"), "CamelCase field is snake_case in python"
def test_message_casing():
assert hasattr(
casing, "SnakeCaseMessage"
), "snake_case Message name is converted to CamelCase in python"
def test_enum_casing():
assert hasattr(
casing, "MyEnum"
), "snake_case Enum name is converted to CamelCase in python"

View File

@@ -0,0 +1,21 @@
syntax = "proto3";
import "google/protobuf/wrappers.proto";
// Tests that wrapped values can be used directly as return values
service Test {
rpc GetDouble (Input) returns (google.protobuf.DoubleValue);
rpc GetFloat (Input) returns (google.protobuf.FloatValue);
rpc GetInt64 (Input) returns (google.protobuf.Int64Value);
rpc GetUInt64 (Input) returns (google.protobuf.UInt64Value);
rpc GetInt32 (Input) returns (google.protobuf.Int32Value);
rpc GetUInt32 (Input) returns (google.protobuf.UInt32Value);
rpc GetBool (Input) returns (google.protobuf.BoolValue);
rpc GetString (Input) returns (google.protobuf.StringValue);
rpc GetBytes (Input) returns (google.protobuf.BytesValue);
}
message Input {
}

View File

@@ -0,0 +1,56 @@
from typing import Any, Callable, Optional
import google.protobuf.wrappers_pb2 as wrappers
import pytest
from betterproto.tests.mocks import MockChannel
from betterproto.tests.output_betterproto.googletypes_response.googletypes_response import (
TestStub,
)
test_cases = [
(TestStub.get_double, wrappers.DoubleValue, 2.5),
(TestStub.get_float, wrappers.FloatValue, 2.5),
(TestStub.get_int64, wrappers.Int64Value, -64),
(TestStub.get_u_int64, wrappers.UInt64Value, 64),
(TestStub.get_int32, wrappers.Int32Value, -32),
(TestStub.get_u_int32, wrappers.UInt32Value, 32),
(TestStub.get_bool, wrappers.BoolValue, True),
(TestStub.get_string, wrappers.StringValue, "string"),
(TestStub.get_bytes, wrappers.BytesValue, bytes(0xFF)[0:4]),
]
@pytest.mark.asyncio
@pytest.mark.parametrize(["service_method", "wrapper_class", "value"], test_cases)
async def test_channel_receives_wrapped_type(
service_method: Callable[[TestStub], Any], wrapper_class: Callable, value
):
wrapped_value = wrapper_class()
wrapped_value.value = value
channel = MockChannel(responses=[wrapped_value])
service = TestStub(channel)
await service_method(service)
assert channel.requests[0]["response_type"] != Optional[type(value)]
assert channel.requests[0]["response_type"] == type(wrapped_value)
@pytest.mark.asyncio
@pytest.mark.xfail
@pytest.mark.parametrize(["service_method", "wrapper_class", "value"], test_cases)
async def test_service_unwraps_response(
service_method: Callable[[TestStub], Any], wrapper_class: Callable, value
):
"""
grpclib does not unwrap wrapper values returned by services
"""
wrapped_value = wrapper_class()
wrapped_value.value = value
service = TestStub(MockChannel(responses=[wrapped_value]))
response_value = await service_method(service)
assert response_value == value
assert type(response_value) == type(value)

View File

@@ -0,0 +1,24 @@
syntax = "proto3";
import "google/protobuf/wrappers.proto";
// Tests that wrapped values are supported as part of output message
service Test {
rpc getOutput (Input) returns (Output);
}
message Input {
}
message Output {
google.protobuf.DoubleValue double_value = 1;
google.protobuf.FloatValue float_value = 2;
google.protobuf.Int64Value int64_value = 3;
google.protobuf.UInt64Value uint64_value = 4;
google.protobuf.Int32Value int32_value = 5;
google.protobuf.UInt32Value uint32_value = 6;
google.protobuf.BoolValue bool_value = 7;
google.protobuf.StringValue string_value = 8;
google.protobuf.BytesValue bytes_value = 9;
}

View File

@@ -0,0 +1,39 @@
import pytest
from betterproto.tests.mocks import MockChannel
from betterproto.tests.output_betterproto.googletypes_response_embedded.googletypes_response_embedded import (
Output,
TestStub,
)
@pytest.mark.asyncio
async def test_service_passes_through_unwrapped_values_embedded_in_response():
"""
We do not not need to implement value unwrapping for embedded well-known types,
as this is already handled by grpclib. This test merely shows that this is the case.
"""
output = Output(
double_value=10.0,
float_value=12.0,
int64_value=-13,
uint64_value=14,
int32_value=-15,
uint32_value=16,
bool_value=True,
string_value="string",
bytes_value=bytes(0xFF)[0:4],
)
service = TestStub(MockChannel(responses=[output]))
response = await service.get_output()
assert response.double_value == 10.0
assert response.float_value == 12.0
assert response.int64_value == -13
assert response.uint64_value == 14
assert response.int32_value == -15
assert response.uint32_value == 16
assert response.bool_value
assert response.string_value == "string"
assert response.bytes_value == bytes(0xFF)[0:4]

View File

@@ -0,0 +1,7 @@
syntax = "proto3";
package package.childpackage;
message ChildMessage {
}

View File

@@ -0,0 +1,9 @@
syntax = "proto3";
import "package_message.proto";
// Tests generated imports when a message in a package refers to a message in a nested child package.
message Test {
package.PackageMessage message = 1;
}

View File

@@ -0,0 +1,9 @@
syntax = "proto3";
import "child.proto";
package package;
message PackageMessage {
package.childpackage.ChildMessage c = 1;
}

View File

@@ -0,0 +1,7 @@
syntax = "proto3";
package childpackage;
message Message {
}

View File

@@ -0,0 +1,9 @@
syntax = "proto3";
import "child.proto";
// Tests generated imports when a message in root refers to a message in a child package.
message Test {
childpackage.Message child = 1;
}

View File

@@ -0,0 +1,28 @@
syntax = "proto3";
import "root.proto";
import "other.proto";
// This test-case verifies that future implementations will support circular dependencies in the generated python files.
//
// This becomes important when generating 1 python file/module per package, rather than 1 file per proto file.
//
// Scenario:
//
// The proto messages depend on each other in a non-circular way:
//
// Test -------> RootPackageMessage <--------------.
// `------------------------------------> OtherPackageMessage
//
// Test and RootPackageMessage are in different files, but belong to the same package (root):
//
// (Test -------> RootPackageMessage) <------------.
// `------------------------------------> OtherPackageMessage
//
// After grouping the packages into single files or modules, a circular dependency is created:
//
// (root: Test & RootPackageMessage) <-------> (other: OtherPackageMessage)
message Test {
RootPackageMessage message = 1;
other.OtherPackageMessage other =2;
}

View File

@@ -0,0 +1,8 @@
syntax = "proto3";
import "root.proto";
package other;
message OtherPackageMessage {
RootPackageMessage rootPackageMessage = 1;
}

View File

@@ -0,0 +1,5 @@
syntax = "proto3";
message RootPackageMessage {
}

View File

@@ -0,0 +1,12 @@
syntax = "proto3";
import "parent_package_message.proto";
package parent.child;
// Tests generated imports when a message refers to a message defined in its parent package
message Test {
ParentPackageMessage message_implicit = 1;
parent.ParentPackageMessage message_explicit = 2;
}

View File

@@ -0,0 +1,6 @@
syntax = "proto3";
package parent;
message ParentPackageMessage {
}

View File

@@ -0,0 +1,11 @@
syntax = "proto3";
import "root.proto";
package child;
// Tests generated imports when a message inside a child-package refers to a message defined in the root.
message Test {
RootMessage message = 1;
}

View File

@@ -0,0 +1,5 @@
syntax = "proto3";
message RootMessage {
}

View File

@@ -0,0 +1,9 @@
syntax = "proto3";
import "sibling.proto";
// Tests generated imports when a message in the root package refers to another message in the root package
message Test {
SiblingMessage sibling = 1;
}

View File

@@ -0,0 +1,5 @@
syntax = "proto3";
message SiblingMessage {
}

View File

@@ -0,0 +1,4 @@
{
"positive": 150,
"negative": -150
}

View File

@@ -3,5 +3,6 @@ syntax = "proto3";
// Some documentation about the Test message.
message Test {
// Some documentation about the count.
int32 count = 1;
int32 positive = 1;
int32 negative = 2;
}

View File

@@ -5,3 +5,7 @@ message Test {
int32 with = 2;
int32 as = 3;
}
service TestService {
rpc GetTest(Test) returns (Test) {}
}

View File

@@ -0,0 +1,3 @@
{
"name": "foobar"
}

View File

@@ -0,0 +1,3 @@
{
"count": 100
}

View File

@@ -0,0 +1,15 @@
import betterproto
from betterproto.tests.output_betterproto.oneof.oneof import Test
from betterproto.tests.util import get_test_case_json_data
def test_which_count():
message = Test()
message.from_json(get_test_case_json_data("oneof"))
assert betterproto.which_one_of(message, "foo") == ("count", 100)
def test_which_name():
message = Test()
message.from_json(get_test_case_json_data("oneof", "oneof-name.json"))
assert betterproto.which_one_of(message, "foo") == ("name", "foobar")

View File

@@ -0,0 +1,3 @@
{
"signal": "PASS"
}

View File

@@ -0,0 +1,3 @@
{
"signal": "RESIGN"
}

View File

@@ -0,0 +1,6 @@
{
"move": {
"x": 2,
"y": 3
}
}

View File

@@ -0,0 +1,18 @@
syntax = "proto3";
message Test {
oneof action {
Signal signal = 1;
Move move = 2;
}
}
enum Signal {
PASS = 0;
RESIGN = 1;
}
message Move {
int32 x = 1;
int32 y = 2;
}

View File

@@ -0,0 +1,42 @@
import pytest
import betterproto
from betterproto.tests.output_betterproto.oneof_enum.oneof_enum import (
Move,
Signal,
Test,
)
from betterproto.tests.util import get_test_case_json_data
@pytest.mark.xfail
def test_which_one_of_returns_enum_with_default_value():
"""
returns first field when it is enum and set with default value
"""
message = Test()
message.from_json(get_test_case_json_data("oneof_enum", "oneof_enum-enum-0.json"))
assert message.move is None
assert message.signal == Signal.PASS
assert betterproto.which_one_of(message, "action") == ("signal", Signal.PASS)
@pytest.mark.xfail
def test_which_one_of_returns_enum_with_non_default_value():
"""
returns first field when it is enum and set with non default value
"""
message = Test()
message.from_json(get_test_case_json_data("oneof_enum", "oneof_enum-enum-1.json"))
assert message.move is None
assert message.signal == Signal.PASS
assert betterproto.which_one_of(message, "action") == ("signal", Signal.RESIGN)
@pytest.mark.xfail
def test_which_one_of_returns_second_field_when_set():
message = Test()
message.from_json(get_test_case_json_data("oneof_enum"))
assert message.move == Move(x=2, y=3)
assert message.signal == 0
assert betterproto.which_one_of(message, "action") == ("move", Move(x=2, y=3))

View File

@@ -0,0 +1,11 @@
syntax = "proto3";
package repeatedmessage;
message Test {
repeated Sub greetings = 1;
}
message Sub {
string greeting = 1;
}

View File

@@ -0,0 +1,15 @@
syntax = "proto3";
package service;
message DoThingRequest {
int32 iterations = 1;
}
message DoThingResponse {
int32 successfulIterations = 1;
}
service Test {
rpc DoThing (DoThingRequest) returns (DoThingResponse);
}

View File

@@ -0,0 +1,132 @@
import betterproto
import grpclib
from grpclib.testing import ChannelFor
import pytest
from typing import Dict
from betterproto.tests.output_betterproto.service.service import (
DoThingResponse,
DoThingRequest,
TestStub as ExampleServiceStub,
)
class ExampleService:
def __init__(self, test_hook=None):
# This lets us pass assertions to the servicer ;)
self.test_hook = test_hook
async def DoThing(
self, stream: "grpclib.server.Stream[DoThingRequest, DoThingResponse]"
):
request = await stream.recv_message()
print("self.test_hook", self.test_hook)
if self.test_hook is not None:
self.test_hook(stream)
for iteration in range(request.iterations):
pass
await stream.send_message(DoThingResponse(request.iterations))
def __mapping__(self) -> Dict[str, grpclib.const.Handler]:
return {
"/service.Test/DoThing": grpclib.const.Handler(
self.DoThing,
grpclib.const.Cardinality.UNARY_UNARY,
DoThingRequest,
DoThingResponse,
),
}
async def _test_stub(stub, iterations=42, **kwargs):
response = await stub.do_thing(iterations=iterations)
assert response.successful_iterations == iterations
def _get_server_side_test(deadline, metadata):
def server_side_test(stream):
assert stream.deadline._timestamp == pytest.approx(
deadline._timestamp, 1
), "The provided deadline should be recieved serverside"
assert (
stream.metadata["authorization"] == metadata["authorization"]
), "The provided authorization metadata should be recieved serverside"
return server_side_test
@pytest.mark.asyncio
async def test_simple_service_call():
async with ChannelFor([ExampleService()]) as channel:
await _test_stub(ExampleServiceStub(channel))
@pytest.mark.asyncio
async def test_service_call_with_upfront_request_params():
# Setting deadline
deadline = grpclib.metadata.Deadline.from_timeout(22)
metadata = {"authorization": "12345"}
async with ChannelFor(
[ExampleService(test_hook=_get_server_side_test(deadline, metadata))]
) as channel:
await _test_stub(
ExampleServiceStub(channel, deadline=deadline, metadata=metadata)
)
# Setting timeout
timeout = 99
deadline = grpclib.metadata.Deadline.from_timeout(timeout)
metadata = {"authorization": "12345"}
async with ChannelFor(
[ExampleService(test_hook=_get_server_side_test(deadline, metadata))]
) as channel:
await _test_stub(
ExampleServiceStub(channel, timeout=timeout, metadata=metadata)
)
@pytest.mark.asyncio
async def test_service_call_lower_level_with_overrides():
ITERATIONS = 99
# Setting deadline
deadline = grpclib.metadata.Deadline.from_timeout(22)
metadata = {"authorization": "12345"}
kwarg_deadline = grpclib.metadata.Deadline.from_timeout(28)
kwarg_metadata = {"authorization": "12345"}
async with ChannelFor(
[ExampleService(test_hook=_get_server_side_test(deadline, metadata))]
) as channel:
stub = ExampleServiceStub(channel, deadline=deadline, metadata=metadata)
response = await stub._unary_unary(
"/service.Test/DoThing",
DoThingRequest(ITERATIONS),
DoThingResponse,
deadline=kwarg_deadline,
metadata=kwarg_metadata,
)
assert response.successful_iterations == ITERATIONS
# Setting timeout
timeout = 99
deadline = grpclib.metadata.Deadline.from_timeout(timeout)
metadata = {"authorization": "12345"}
kwarg_timeout = 9000
kwarg_deadline = grpclib.metadata.Deadline.from_timeout(kwarg_timeout)
kwarg_metadata = {"authorization": "09876"}
async with ChannelFor(
[
ExampleService(
test_hook=_get_server_side_test(kwarg_deadline, kwarg_metadata)
)
]
) as channel:
stub = ExampleServiceStub(channel, deadline=deadline, metadata=metadata)
response = await stub._unary_unary(
"/service.Test/DoThing",
DoThingRequest(ITERATIONS),
DoThingResponse,
timeout=kwarg_timeout,
metadata=kwarg_metadata,
)
assert response.successful_iterations == ITERATIONS

View File

@@ -0,0 +1,6 @@
{
"signed32": 150,
"negative32": -150,
"string64": "150",
"negative64": "-150"
}

View File

@@ -0,0 +1,9 @@
syntax = "proto3";
message Test {
// todo: rename fields after fixing bug where 'signed_32_positive' will map to 'signed_32Positive' as output json
sint32 signed32 = 1; // signed_32_positive
sint32 negative32 = 2; // signed_32_negative
sint64 string64 = 3; // signed_64_positive
sint64 negative64 = 4; // signed_64_negative
}

View File

@@ -0,0 +1,10 @@
# Test cases that are expected to fail, e.g. unimplemented features or bug-fixes.
# Remove from list when fixed.
tests = {
"import_root_sibling",
"import_child_package_from_package",
"import_root_package_from_child",
"import_parent_package_from_child",
"import_circular_dependency",
"oneof_enum",
}

View File

@@ -1,3 +0,0 @@
{
"count": -150
}

View File

@@ -1,3 +0,0 @@
{
"count": 150
}

View File

@@ -0,0 +1,39 @@
from typing import List
from grpclib.client import Channel
class MockChannel(Channel):
# noinspection PyMissingConstructor
def __init__(self, responses=None) -> None:
self.responses = responses if responses else []
self.requests = []
def request(self, route, cardinality, request, response_type, **kwargs):
self.requests.append(
{
"route": route,
"cardinality": cardinality,
"request": request,
"response_type": response_type,
}
)
return MockStream(self.responses)
class MockStream:
def __init__(self, responses: List) -> None:
super().__init__()
self.responses = responses
async def recv_message(self):
return self.responses.pop(0)
async def send_message(self, *args, **kwargs):
pass
async def __aexit__(self, exc_type, exc_val, exc_tb):
return True
async def __aenter__(self):
return self

View File

@@ -1,3 +0,0 @@
{
"name": "foo"
}

View File

@@ -1,3 +0,0 @@
{
"count": 1
}

View File

@@ -1,4 +0,0 @@
{
"signed_32": -150,
"signed_64": "-150"
}

View File

@@ -1,4 +0,0 @@
{
"signed_32": 150,
"signed_64": "150"
}

View File

@@ -1,6 +0,0 @@
syntax = "proto3";
message Test {
sint32 signed_32 = 1;
sint64 signed_64 = 2;
}

View File

@@ -256,7 +256,7 @@ def test_to_dict_default_values():
some_double: float = betterproto.double_field(2)
some_message: TestChildMessage = betterproto.message_field(3)
test = TestParentMessage().from_dict({"someInt": 0, "someDouble": 1.2,})
test = TestParentMessage().from_dict({"someInt": 0, "someDouble": 1.2})
assert test.to_dict(include_default_values=True) == {
"someInt": 0,

View File

@@ -1,32 +1,147 @@
import importlib
import json
import os
import sys
from collections import namedtuple
from typing import Set
import pytest
from .generate import get_base, get_files
import betterproto
from betterproto.tests.inputs import xfail
from betterproto.tests.mocks import MockChannel
from betterproto.tests.util import get_directories, get_test_case_json_data, inputs_path
inputs = get_files(".bin")
# Force pure-python implementation instead of C++, otherwise imports
# break things because we can't properly reset the symbol database.
os.environ["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"] = "python"
from google.protobuf import symbol_database
from google.protobuf.descriptor_pool import DescriptorPool
from google.protobuf.json_format import Parse
@pytest.mark.parametrize("filename", inputs)
def test_sample(filename: str) -> None:
module = get_base(filename).split("-")[0]
imported = importlib.import_module(f"betterproto.tests.{module}")
data_binary = open(filename, "rb").read()
data_dict = json.loads(open(filename.replace(".bin", ".json")).read())
t1 = imported.Test().parse(data_binary)
t2 = imported.Test().from_dict(data_dict)
print(t1)
print(t2)
class TestCases:
def __init__(self, path, services: Set[str], xfail: Set[str]):
_all = set(get_directories(path))
_services = services
_messages = _all - services
_messages_with_json = {
test for test in _messages if get_test_case_json_data(test)
}
# Equality should automagically work for dataclasses!
assert t1 == t2
self.all = self.apply_xfail_marks(_all, xfail)
self.services = self.apply_xfail_marks(_services, xfail)
self.messages = self.apply_xfail_marks(_messages, xfail)
self.messages_with_json = self.apply_xfail_marks(_messages_with_json, xfail)
# Generally this can't be relied on, but here we are aiming to match the
# existing Python implementation and aren't doing anything tricky.
# https://developers.google.com/protocol-buffers/docs/encoding#implications
assert bytes(t1) == data_binary
assert bytes(t2) == data_binary
@staticmethod
def apply_xfail_marks(test_set: Set[str], xfail: Set[str]):
return [
pytest.param(test, marks=pytest.mark.xfail) if test in xfail else test
for test in test_set
]
assert t1.to_dict() == data_dict
assert t2.to_dict() == data_dict
test_cases = TestCases(
path=inputs_path,
# test cases for services
services={"googletypes_response", "googletypes_response_embedded", "service"},
xfail=xfail.tests,
)
plugin_output_package = "betterproto.tests.output_betterproto"
reference_output_package = "betterproto.tests.output_reference"
TestData = namedtuple("TestData", "plugin_module, reference_module, json_data")
@pytest.fixture
def test_data(request):
test_case_name = request.param
# Reset the internal symbol database so we can import the `Test` message
# multiple times. Ugh.
sym = symbol_database.Default()
sym.pool = DescriptorPool()
reference_module_root = os.path.join(
*reference_output_package.split("."), test_case_name
)
sys.path.append(reference_module_root)
yield (
TestData(
plugin_module=importlib.import_module(
f"{plugin_output_package}.{test_case_name}.{test_case_name}"
),
reference_module=lambda: importlib.import_module(
f"{reference_output_package}.{test_case_name}.{test_case_name}_pb2"
),
json_data=get_test_case_json_data(test_case_name),
)
)
sys.path.remove(reference_module_root)
@pytest.mark.parametrize("test_data", test_cases.messages, indirect=True)
def test_message_can_instantiated(test_data: TestData) -> None:
plugin_module, *_ = test_data
plugin_module.Test()
@pytest.mark.parametrize("test_data", test_cases.messages, indirect=True)
def test_message_equality(test_data: TestData) -> None:
plugin_module, *_ = test_data
message1 = plugin_module.Test()
message2 = plugin_module.Test()
assert message1 == message2
@pytest.mark.parametrize("test_data", test_cases.messages_with_json, indirect=True)
def test_message_json(repeat, test_data: TestData) -> None:
plugin_module, _, json_data = test_data
for _ in range(repeat):
message: betterproto.Message = plugin_module.Test()
message.from_json(json_data)
message_json = message.to_json(0)
assert json.loads(json_data) == json.loads(message_json)
@pytest.mark.parametrize("test_data", test_cases.services, indirect=True)
def test_service_can_be_instantiated(test_data: TestData) -> None:
plugin_module, _, json_data = test_data
plugin_module.TestStub(MockChannel())
@pytest.mark.parametrize("test_data", test_cases.messages_with_json, indirect=True)
def test_binary_compatibility(repeat, test_data: TestData) -> None:
plugin_module, reference_module, json_data = test_data
reference_instance = Parse(json_data, reference_module().Test())
reference_binary_output = reference_instance.SerializeToString()
for _ in range(repeat):
plugin_instance_from_json: betterproto.Message = plugin_module.Test().from_json(
json_data
)
plugin_instance_from_binary = plugin_module.Test.FromString(
reference_binary_output
)
# # Generally this can't be relied on, but here we are aiming to match the
# # existing Python implementation and aren't doing anything tricky.
# # https://developers.google.com/protocol-buffers/docs/encoding#implications
assert bytes(plugin_instance_from_json) == reference_binary_output
assert bytes(plugin_instance_from_binary) == reference_binary_output
assert plugin_instance_from_json == plugin_instance_from_binary
assert (
plugin_instance_from_json.to_dict() == plugin_instance_from_binary.to_dict()
)

61
betterproto/tests/util.py Normal file
View File

@@ -0,0 +1,61 @@
import os
import subprocess
from typing import Generator
os.environ["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"] = "python"
root_path = os.path.dirname(os.path.realpath(__file__))
inputs_path = os.path.join(root_path, "inputs")
output_path_reference = os.path.join(root_path, "output_reference")
output_path_betterproto = os.path.join(root_path, "output_betterproto")
if os.name == "nt":
plugin_path = os.path.join(root_path, "..", "plugin.bat")
else:
plugin_path = os.path.join(root_path, "..", "plugin.py")
def get_files(path, end: str) -> Generator[str, None, None]:
for r, dirs, files in os.walk(path):
for filename in [f for f in files if f.endswith(end)]:
yield os.path.join(r, filename)
def get_directories(path):
for root, directories, files in os.walk(path):
for directory in directories:
yield directory
def relative(file: str, path: str):
return os.path.join(os.path.dirname(file), path)
def read_relative(file: str, path: str):
with open(relative(file, path)) as fh:
return fh.read()
def protoc_plugin(path: str, output_dir: str):
subprocess.run(
f"protoc --plugin=protoc-gen-custom={plugin_path} --custom_out={output_dir} --proto_path={path} {path}/*.proto",
shell=True,
)
def protoc_reference(path: str, output_dir: str):
subprocess.run(
f"protoc --python_out={output_dir} --proto_path={path} {path}/*.proto",
shell=True,
)
def get_test_case_json_data(test_case_name, json_file_name=None):
test_data_file_name = json_file_name if json_file_name else f"{test_case_name}.json"
test_data_file_path = os.path.join(inputs_path, test_case_name, test_data_file_name)
if not os.path.exists(test_data_file_path):
return None
with open(test_data_file_path) as fh:
return fh.read()

10
conftest.py Normal file
View File

@@ -0,0 +1,10 @@
import pytest
def pytest_addoption(parser):
parser.addoption("--repeat", type=int, default=1, help="repeat the operation multiple times")
@pytest.fixture(scope="session")
def repeat(request):
return request.config.getoption("repeat")

5
pytest.ini Normal file
View File

@@ -0,0 +1,5 @@
[pytest]
python_files = test_*.py
python_classes =
norecursedirs = **/output_*
addopts = -p no:warnings

View File

@@ -2,9 +2,9 @@ from setuptools import setup, find_packages
setup(
name="betterproto",
version="1.2.3",
version="1.2.4",
description="A better Protobuf / gRPC generator & library",
long_description=open("README.md", "r").read(),
long_description=open("README.md", "r", encoding="utf-8").read(),
long_description_content_type="text/markdown",
url="http://github.com/danielgtaylor/python-betterproto",
author="Daniel G. Taylor",