183 Commits

Author SHA1 Message Date
boukeversteegh
0c02d1b21a Update with master 2020-07-04 18:54:26 +02:00
Bouke Versteegh
ac32bcd25a Merge branch 'master' into michael-sayapin/master 2020-07-04 11:23:42 +02:00
Bouke Versteegh
cdddb2f42a Merge pull request #88 from boukeversteegh/fix/imports
🍏 Fix imports
2020-07-04 11:22:12 +02:00
boukeversteegh
d21cd6e391 black 2020-07-01 13:15:03 +02:00
boukeversteegh
af7115429a Expose betterproto.ServiceStub 2020-07-01 12:43:28 +02:00
boukeversteegh
0d9387abec Remove stringcase dependency 2020-07-01 12:43:12 +02:00
boukeversteegh
f4ebcb0f65 Merge remote-tracking branch 'daniel/master' into fix/imports
# Conflicts:
#	Pipfile
#	README.md
#	betterproto/__init__.py
#	betterproto/plugin.py
#	betterproto/tests/util.py
2020-07-01 12:19:25 +02:00
boukeversteegh
81711d2427 Avoid naming conflicts when importing multiple types with the same name from an ancestor package 2020-07-01 12:07:59 +02:00
boukeversteegh
e3135ce766 Add parameter for non-strict cased output that preserves delimiter count 2020-07-01 09:39:37 +02:00
boukeversteegh
72855227bd Fix import 2020-06-25 15:52:43 +02:00
boukeversteegh
47081617c2 Merge branch 'master' into michael-sayapin/master 2020-06-25 15:02:50 +02:00
Bouke Versteegh
9532844929 Merge pull request #83 from nat-n/client-streaming
Client streaming
2020-06-24 22:13:54 +02:00
boukeversteegh
d734206fe5 Rename test-case to keep it close with other enum test 2020-06-24 21:55:31 +02:00
Bouke Versteegh
bbf40f9694 Mark test xfail 2020-06-24 21:48:26 +02:00
nat
0c5d1ff868 Merge branch 'master' into client-streaming 2020-06-23 22:02:23 +02:00
Bouke Versteegh
5fb4b4b7ff Merge pull request #75 from nat-n/add_poetry
Switch from pipenv to poetry
2020-06-23 21:59:46 +02:00
Nat Noordanus
4f820b4a6a Include python 3.8 i ci test runs & optimise CI and make config 2020-06-22 19:38:41 +02:00
Nat Noordanus
75a4c230da Add optional deps to dev-deps
So contributors dont have to remember to run poetry install with `-E compiler`
2020-06-22 19:35:23 +02:00
Michael Sayapin
6671d87cef Conformance formatting 2020-06-17 11:37:36 +08:00
nat
5c9a12e2f6 Merge pull request #1 from boukeversteegh/client-streaming-tests
Client streaming tests
2020-06-16 19:36:40 +02:00
Nat Noordanus
e1ccd540a9 Fix bugs and remove footgun feature in AsyncChannel 2020-06-16 00:07:28 +02:00
nat
4e78fe9579 Merge branch 'client-streaming' into client-streaming-tests 2020-06-15 23:42:01 +02:00
Nat Noordanus
50bb67bf5d Fix bugs and remove footgun feature in AsyncChannel 2020-06-15 23:35:56 +02:00
Bouke Versteegh
1ecbf1a125 Merge pull request #90 from jameslan/fix/fixed-types
fixed field types should be int
2020-06-15 19:48:31 +02:00
boukeversteegh
0814729c5a Add cases for send() 2020-06-15 18:14:13 +02:00
boukeversteegh
f7aa6150e2 Add test-cases for client stream-stream 2020-06-15 18:02:37 +02:00
boukeversteegh
159c30ddd8 Fix close not awaitable, fix done is callable, fix return async next value 2020-06-15 18:02:05 +02:00
Michael Sayapin
cd66b0511a Fixes enum class name 2020-06-15 13:52:58 +08:00
Michael Sayapin
c48ca2e386 Test to_dict with missing enum values 2020-06-15 12:51:51 +08:00
Nat Noordanus
c8229e53a7 Fix most mypy warnings 2020-06-15 00:19:07 +02:00
Nat Noordanus
3185c67098 Improve generate script
- Fix issue with __pycache__ dirs getting picked up
- parallelise code generation with asyncio for 3x speedup
- silence protoc output unless -v option is supplied
- Use pathlib ;)
2020-06-15 00:19:07 +02:00
boukeversteegh
52eea5ce4c Added missing tests for casing 2020-06-14 23:15:56 +02:00
Nat Noordanus
4b6f55dce5 Finish implementation and testing of client
Including stream_unary and stream_stream call methods.

Also
- improve organisation of relevant tests
- fix some generated type annotations
- Add AsyncChannel utility cos it's useful
2020-06-14 23:04:52 +02:00
boukeversteegh
fdbe0205f1 find_module docstring and search for init files instead of directories 2020-06-14 22:54:03 +02:00
Nat Noordanus
09f821921f Move ServiceStub to a seperate module and add more rpcs to service test 2020-06-14 22:19:51 +02:00
Hans Lellelid
a757da1b29 Adding basic support (untested) for client streaming 2020-06-14 22:19:51 +02:00
boukeversteegh
e2d672a422 Fix terminology, improve docstrings and add missing asserts to tests 2020-06-14 21:40:12 +02:00
boukeversteegh
63f5191f02 Shorten list selectors 2020-06-14 16:54:34 +02:00
boukeversteegh
87f4b34930 Revert "Support running plugin without installing betterproto"
This reverts commit c88edfd0
2020-06-14 16:52:33 +02:00
boukeversteegh
2c360a55f2 Readability for generating init_files 2020-06-14 16:51:52 +02:00
James Lan
04dce524aa fixed field types should be int 2020-06-12 17:04:56 -07:00
Nat Noordanus
8edec81b11 Switch from pipenv to poetry
- dropped dev dependency on rope, isort & flake
- poetry doesn't support dev scripts like pipenv, so create a makefile instead
- Add pytest-cov
- Use tox for testing multiple python versions in CI
- Update README

Update ci workflow
2020-06-12 21:13:55 +02:00
boukeversteegh
32c8e77274 Recompile Google Protobuf files 2020-06-12 13:56:32 +02:00
boukeversteegh
d9fa6d2dd3 Fixes issue where generated Google Protobuf messages imported from betterproto.lib instead of using local forward references 2020-06-12 13:55:55 +02:00
boukeversteegh
c88edfd093 Support running plugin without installing betterproto 2020-06-12 13:54:14 +02:00
nat
a46979c8a6 Merge pull request #86 from danielgtaylor/boukeversteegh-patch-1
Add Slack invite link
2020-06-11 17:26:38 +02:00
boukeversteegh
83e13aa606 Fix method name 2020-06-11 13:55:12 +02:00
boukeversteegh
3ca75dadd7 Remove dependency on stringcase, apply black 2020-06-11 13:55:12 +02:00
boukeversteegh
5d2f3a2cd9 Remove fixed test from xfail list #11 2020-06-11 13:55:12 +02:00
boukeversteegh
65c1f366ef Update readme with new output structure and fix example inconsistencies 2020-06-11 13:55:12 +02:00
boukeversteegh
34c34bd15a Add failing test for importing a message from package that looks like a nested type #87 2020-06-11 13:55:12 +02:00
boukeversteegh
fb54917f2c Detect entry-point of tests automatically 2020-06-11 13:55:12 +02:00
boukeversteegh
1a95a7988e Ensure uniquely generated import aliases are not name mangled (python.org/dev/peps/pep-0008/#id34) 2020-06-11 13:55:11 +02:00
boukeversteegh
76db2f153e Add import aliases to ancestor imports 2020-06-11 13:55:11 +02:00
boukeversteegh
8567892352 Simplify logic for generating package init files 2020-06-11 13:55:11 +02:00
boukeversteegh
3105e952ea Fixes issue where importing cousin where path has a package with the same name broke import 2020-06-11 13:55:11 +02:00
boukeversteegh
7c8d47de6d Add test cases for cousin imports that break due to aliases starting with two underscores 2020-06-11 13:55:11 +02:00
boukeversteegh
c00e2aef19 Break up importing logic in methods 2020-06-11 13:55:11 +02:00
boukeversteegh
fdf3b2e764 Compile proto files based on package structure 2020-06-11 13:55:11 +02:00
boukeversteegh
f7c2fd1194 Support nested messages, fix casing. Support test-cases in packages. 2020-06-11 13:55:11 +02:00
boukeversteegh
d8abb850f8 Update tests to reflect new generated package structure 2020-06-11 13:55:11 +02:00
boukeversteegh
d7ba27de2b fix all broken imports 2020-06-11 13:55:11 +02:00
boukeversteegh
57523a9e7f Implement importing unrelated package 2020-06-11 13:55:11 +02:00
boukeversteegh
e5e61c873c Implement some import scenarios 2020-06-11 13:55:11 +02:00
boukeversteegh
9fd1c058e6 Create unit tests for importing 2020-06-11 13:55:11 +02:00
boukeversteegh
d336153845 Use never expiring invitation link 2020-06-11 13:49:53 +02:00
nat
9a45ea9f16 Merge pull request #78 from boukeversteegh/pr/google
Basic general support for Google Protobuf
2020-06-11 10:50:12 +02:00
Bouke Versteegh
bb7f5229fb Add Slack invite link 2020-06-10 17:30:18 +02:00
boukeversteegh
f7769a19d1 Pass betterproto option using custom_opt instead of environment variable 2020-06-06 12:51:37 +02:00
boukeversteegh
d31f90be6b Combine circular imports 2020-06-04 00:11:22 +02:00
boukeversteegh
919b0a6a7d Check if betterproto has wrapper support in idiomatic way 2020-06-04 00:02:28 +02:00
boukeversteegh
7ecf3fe0e6 Add comment to explain unusual import location 2020-06-04 00:02:28 +02:00
Bouke Versteegh
ff14948a4e Use raw string for regex
Co-authored-by: nat <nat.noordanus@gmail.com>
2020-06-04 00:02:28 +02:00
Bouke Versteegh
cb00273257 Fix name PROTOBUF_OPTS -> BETTERPROTO_OPTS 2020-06-04 00:02:28 +02:00
boukeversteegh
973d68a154 Add missing field to MockChannel to prevent warnings while testing 2020-06-04 00:02:28 +02:00
boukeversteegh
ab9857b5fd Add test-case for service that returns google protobuf values 2020-06-04 00:02:28 +02:00
boukeversteegh
2f658df666 Use betterproto wrapper classes, extract to module for testability 2020-06-04 00:02:28 +02:00
boukeversteegh
b813d1cedb Undo adding skip to test 2020-06-03 23:59:10 +02:00
boukeversteegh
f5ce1b7108 Check that config.xfail contains valid test case names 2020-06-03 23:59:10 +02:00
boukeversteegh
62fc421d60 Add failing tests for google.protobuf Struct and Value #9 2020-06-03 23:59:10 +02:00
boukeversteegh
eeed1c0db7 Extend pre-compiled Duration and Timestamp instead of manual definition 2020-06-03 23:58:47 +02:00
boukeversteegh
2a3e1e1827 Add basic support for all google.protobuf types 2020-06-03 23:58:47 +02:00
boukeversteegh
53ce1255d3 Do not unwrap google.protobuf.Value and unsupported wrapper types 2020-06-03 23:58:47 +02:00
boukeversteegh
e8991339e9 Use pre-compiled wrapper-classes 2020-06-03 23:54:43 +02:00
boukeversteegh
4556d67503 Include pre-compiled google protobuf classes 2020-06-03 23:54:43 +02:00
boukeversteegh
f087c6c9bd Support compiling google protobuf files 2020-06-03 23:54:43 +02:00
Bouke Versteegh
eec24e4ee8 Merge pull request #77 from danielgtaylor/nat-n-patch-1
Rearrange plugin import to make import errors more helpful
2020-05-30 20:52:35 +02:00
nat
91111ab7d8 Make plugin import errors more helpful
This addresses an issue where if the user happens to have black installed in
their environment but not the other dependencies when running the protoc
plugin then the resulting import error (No module named 'google') is not very
helpful.
2020-05-30 16:08:36 +02:00
Bouke Versteegh
fcff3dff74 Merge pull request #62 from jameslan/perf/cache-fields
Cache field metadata, to avoid calling `dataclasses.fields` to get more than 10% performance improvement
2020-05-29 12:17:25 +02:00
Bouke Versteegh
5c4969ff1c Merge pull request #69 from boukeversteegh/pr/bugreports
Bugreports
2020-05-28 09:07:11 +02:00
James Lan
ed33a48d64 Cache field metadata, to avoid calling dataclasses.fields to get more than 10% performance improvement 2020-05-27 15:58:14 -07:00
nat
ee362a7a73 Merge pull request #73 from nat-n/always_black
Bump version to 1.2.5
2020-05-27 13:37:54 +02:00
nat
261e55b2c8 Merge pull request #72 from nat-n/always_black
Make CI check formatting is black & append .j2 suffix to template.py
2020-05-27 12:27:33 +02:00
Nat Noordanus
98930ce0d7 Bump version to 1.2.5 2020-05-27 12:04:53 +02:00
Nat Noordanus
d7d277eb0d Remove typo from Pipfile and update Pipfile.lock 2020-05-27 11:52:18 +02:00
Nat Noordanus
3860c0ab11 Add task to run black --check in ci & update README 2020-05-27 11:52:10 +02:00
Nat Noordanus
cd1c2dc3b5 Rename template file to avoid confusing black or other build tools 2020-05-27 11:25:19 +02:00
Nat Noordanus
be2a24d15c blacken 2020-05-27 11:25:00 +02:00
Vasilios Syrakis
a5effb219a Release 1.2.4 (#71)
Co-authored-by: nat <n@natn.me>
2020-05-26 22:17:55 +02:00
boukeversteegh
b354aeb692 Add dict to list of built-types for #53 2020-05-26 10:09:58 +02:00
boukeversteegh
6d9e3fc580 Add issue references to failing test cases 2020-05-25 23:43:01 +02:00
boukeversteegh
72de590651 Remove unused proto file 2020-05-25 23:36:09 +02:00
boukeversteegh
3c70f21074 #70 Messages should allow fields that are Python keywords 2020-05-25 23:36:08 +02:00
boukeversteegh
4b7d5d3de4 #53 Crash when field has the same name as a system type 2020-05-25 22:23:39 +02:00
Bouke Versteegh
2d57f0d122 Merge pull request #67 from danielgtaylor/nat-n-patch-1
Enforce utf-8 for reading the readme in setup.py
2020-05-25 21:57:12 +02:00
boukeversteegh
142e976c40 Add extra related test cases for #11 2020-05-25 21:56:03 +02:00
boukeversteegh
382fabb96c #11 ALL_CAPS message fields are parsed incorrectly 2020-05-25 21:50:30 +02:00
boukeversteegh
18598e77d4 Remove renamed service from test input config 2020-05-25 21:38:14 +02:00
boukeversteegh
6871053ab2 #9 Import bug - returning well known type Empty from service 2020-05-25 21:21:33 +02:00
boukeversteegh
5bb6931df7 #25 Two packages with the same name suffix should not cause naming conflict 2020-05-25 21:15:39 +02:00
boukeversteegh
e8a9960b73 Move configuration of test-cases to config file, include list of service tests 2020-05-25 21:11:33 +02:00
boukeversteegh
f25c66777a #68 Service input messages are not imported 2020-05-25 18:48:42 +02:00
nat
a68505b80e Enforce utf-8 for reading the readme
Fixes failing installation issue #66
2020-05-25 17:53:13 +02:00
nat
2f9497e064 Merge pull request #55 from boukeversteegh/pr/xfail-tests
Add intentionally failing test-cases for unimplemented bug-fixes
2020-05-25 09:54:26 +02:00
boukeversteegh
33964b883e Do not use mutable defaults 2020-05-25 00:35:43 +02:00
boukeversteegh
ec7574086d Add xfail test-case to for future circular dependency scenario 2020-05-24 20:35:10 +02:00
boukeversteegh
8a42027bc9 Improve failing test-case for issue #64 2020-05-24 20:33:48 +02:00
boukeversteegh
71737cf696 Test case for issue #63 2020-05-24 20:29:32 +02:00
boukeversteegh
659ddd9c44 Working test case for oneof 2020-05-24 20:29:19 +02:00
boukeversteegh
5b6997870a Test case for issue #61 2020-05-24 20:27:12 +02:00
boukeversteegh
cdf7645722 Test case for issue #60 2020-05-24 20:26:47 +02:00
boukeversteegh
ca20069ca3 Test case for issue #59 2020-05-24 20:26:13 +02:00
boukeversteegh
59a4a7da43 Test case for issue #58 2020-05-24 20:25:29 +02:00
boukeversteegh
15af4367e5 Test case for issue #57 2020-05-24 20:24:55 +02:00
boukeversteegh
ec5683e572 Test Service instantiation as part of standard test-case 2020-05-24 20:02:41 +02:00
boukeversteegh
20150fdcf3 Cleanup 2020-05-24 19:58:49 +02:00
boukeversteegh
d11b7d04c5 Document XFAIL tests 2020-05-24 19:58:35 +02:00
boukeversteegh
e2d35f4696 Support xfail on test-case level, support running tests on subsets. 2020-05-24 19:58:06 +02:00
boukeversteegh
c3f08b9ef2 Clear output directories before generating python files 2020-05-24 19:54:53 +02:00
boukeversteegh
24d44898f4 Only import reference module when needed. Some reference modules generate bad imports and cannot be loaded. 2020-05-24 19:53:14 +02:00
boukeversteegh
074448c996 Restore accidentally removed binary equality test 2020-05-24 19:52:14 +02:00
nat
0fe557bd3c Merge pull request #52 from nat-n/fix_type_imports
Only import types from grpclib when type checking
2020-05-24 19:09:08 +02:00
nat
1a87ea43a1 Merge pull request #40 from boukeversteegh/pr/wrapper-as-output
Support using Google's wrapper types as RPC output values
2020-05-24 19:06:30 +02:00
andrei
983e0895a2 Fix services using non-pythonified field names 2020-05-24 18:46:36 +02:00
nat
4a2baf3f0a Merge pull request #46 from jameslan/perf/class-cache
Improve performance of serialize/deserialize by caching type information of fields in class
2020-05-24 18:38:32 +02:00
boukeversteegh
8f0caf1db2 Read desired wrapper type directly from wrapper definition 2020-05-24 14:50:56 +02:00
boukeversteegh
c50d9e2fdc Add test for generating embedded wellknown types in outputs. 2020-05-24 14:48:39 +02:00
boukeversteegh
35548cb43e Test all supported wrapper types. Add xfail test for unwrapping the value 2020-05-24 12:34:37 +02:00
boukeversteegh
b711d1e11f Merge remote-tracking branch 'daniel/master' into pr/wrapper-as-output 2020-05-24 10:41:40 +02:00
James Lan
917de09bb6 Replace extra decorator with property and lazy initialization so that it is backward compatible. 2020-05-23 17:36:29 -07:00
James Lan
1f7f39049e Cache resolved classes for fields, so that there's no new data classes generated while deserializing. 2020-05-23 17:36:29 -07:00
James Lan
3d001a2a1a Store the class metadata of fields in the class, to improve preformance
Cached data include,
- lookup table between groups and fields of "oneof" fields
- default value creator of each field
- type hint of each field
2020-05-23 17:36:29 -07:00
James Lan
de61ddab21 Add option to repeatly execute betterproto operations in test, to evaluate performance 2020-05-23 17:36:29 -07:00
Nat Noordanus
5e2d9febea Blacken 2020-05-23 23:37:22 +02:00
nat
f6af077ffe Merge pull request #51 from boukeversteegh/pr/refactor-tests
Reorganize tests and add some extra documentation.
2020-05-22 22:32:37 +02:00
boukeversteegh
92088ebda8 Cleanup 2020-05-22 21:18:44 +02:00
boukeversteegh
c3e3837f71 More concise whitelist logic 2020-05-22 21:11:23 +02:00
boukeversteegh
6bd9c7835c Fix docs 2020-05-22 21:08:08 +02:00
boukeversteegh
6ec902c1b5 Fix generate noargs. Sorted iteration. 2020-05-22 21:03:45 +02:00
boukeversteegh
960dba2ae8 Renamed docs for standard tests 2020-05-22 20:58:53 +02:00
boukeversteegh
4b4bdefb6f Add explicit test for casing rules 2020-05-22 20:58:31 +02:00
boukeversteegh
dfa0a56b39 Simplify standard tests by using 1 json per case. 2020-05-22 20:58:14 +02:00
boukeversteegh
dd4873dfba Re-introducing whitelisting argument to generate.py 2020-05-22 20:51:22 +02:00
Nat Noordanus
91f586f7d7 Apply black formatting 2020-05-22 18:46:43 +02:00
Nat Noordanus
33fb83faad Only import types from grpclib when type checking 2020-05-22 18:41:29 +02:00
boukeversteegh
77c04414f5 Update readme, add docs for standard tests 2020-05-22 16:36:43 +02:00
boukeversteegh
6969ff7ff6 Add another missing gitignored file, and remove gitignore filter for tests/ 2020-05-22 15:34:25 +02:00
boukeversteegh
13e08fdaa8 Add missing file, ignore output files 2020-05-22 15:05:52 +02:00
boukeversteegh
6775632f77 Undo unintentional pipfile update 2020-05-22 13:03:52 +02:00
boukeversteegh
b12f1e4e61 Organize test-cases into folders, extract compatibility test into proper test, support adding test-case specific tests 2020-05-22 12:54:01 +02:00
Bouke Versteegh
7e9ba0866c cleanup 2020-05-21 22:55:26 +02:00
nat
3546f55146 Merge pull request #32 from nat-n/improve_stub
Add ability to provide metadata, timeout & deadline args to requests
2020-05-21 10:11:45 +02:00
boukeversteegh
499489f1d3 Support using Google's wrapper types as RPC output values 2020-05-10 16:36:29 +02:00
Vasili Syrakis
ce9f492f50 Increment version to 1.2.3 2020-04-15 14:24:02 +10:00
Vasilios Syrakis
93a6334015 Update CHANGELOG.md 2020-04-15 14:21:30 +10:00
Adam Ehlers Nyholm Thomsen
36a14026d8 Fix issue that occurs with naming when proto is double nested (#21) 2020-04-15 14:10:43 +10:00
Vasilios Syrakis
04a2fcd3eb Merge pull request #31 from nat-n/fix_readme
Fix test instructions to match pipfile
2020-04-14 10:55:18 +10:00
Nat Noordanus
5759e323bd Add ability to provide metadata, timeout & deadline args to requests
This is an enhancement of the ServiceStub abstract class that makes
it more useful by making it possible to pass all arguments supported
by the underlying grpclib request function.

It extends to the existing high level API by allowing values to be
set on the stub instance, and the low level API by allowing values
to be set per call.
2020-04-12 22:23:10 +02:00
Nat Noordanus
c762c9c549 Add test for generated service stub
- Create one simple test for generated Service stubs in preparation
for making more changes in this area.
- Add dev dependency on pytest-asyncio in order to use ChannelFor
from grpclib.testing more easily.
- Create a new example proto containing a minimal rpc example.
2020-04-12 19:37:39 +02:00
Nat Noordanus
582a12577c Fix test instructions to match pipfile 2020-04-12 18:52:43 +02:00
Vasilios Syrakis
3616190451 Merge pull request #30 from nat-n/p36_support
#27 Add support for python 3.6
2020-04-08 09:37:48 +10:00
Nat Noordanus
9b990ee1bd Make pipenv play nice with the setup-python ci workflow 2020-04-05 15:58:12 +02:00
Vasilios Syrakis
72a77b0d65 Merge pull request #28 from tanishq-dubey/patch-1
Update README.md for pip syntax
2020-04-05 14:52:48 +10:00
Nat Noordanus
b2b36c8575 Apply black formatting 2020-04-03 19:54:19 +02:00
Nat Noordanus
203105f048 Add support for python 3.6
Changes:
- Update config and docs to reference 3.6
- Add backports of dataclasses and datetime.fromisoformat for python_version<"3.7"
- Support both 3.7 and 3.6 usages of undocumented __origin__ attribute on typing objects
- Make github ci run tests for python 3.6 as well
2020-04-03 19:52:19 +02:00
Tanishq Dubey
fe11f74227 Update README.md
Add quotes to the README so pip syntax is correct
2020-03-30 09:50:11 -04:00
Daniel G. Taylor
dc7a3e9bdf Update changelog 2020-01-30 17:48:12 -08:00
Daniel G. Taylor
f2e8afc609 Merge pull request #16 from cetanu/patch-1
Exclude empty lists from to_dict output
2020-01-30 17:31:25 -08:00
Daniel G. Taylor
dbd438e682 Update to emit empty lists if asked for defaults 2020-01-30 17:28:22 -08:00
Daniel G. Taylor
dce1c89fbe Merge branch 'master' into patch-1 2020-01-30 17:22:47 -08:00
Daniel G. Taylor
c78851b1b8 Merge pull request #12 from ulasozguler/master
Added `include_default_values` parameter to `to_dict` function
2020-01-30 17:19:34 -08:00
Vasilios Syrakis
4554d91f89 Exclude empty lists from to_dict output 2020-01-29 22:32:35 +11:00
ulas
c0170f4d80 Added include_default_values parameter to to_dict function. 2020-01-22 19:16:57 +03:00
148 changed files with 5941 additions and 1098 deletions

View File

@@ -3,31 +3,72 @@ name: CI
on: [push, pull_request]
jobs:
build:
check-formatting:
runs-on: ubuntu-latest
name: Consult black on python formatting
steps:
- uses: actions/checkout@v1
- uses: actions/setup-python@v1
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.7
- uses: dschep/install-pipenv-action@v1
- uses: Gr1N/setup-poetry@v2
- uses: actions/cache@v2
with:
path: ~/.cache/pypoetry/virtualenvs
key: ${{ runner.os }}-poetry-${{ hashFiles('poetry.lock') }}
restore-keys: |
${{ runner.os }}-poetry-
- name: Install dependencies
run: poetry install
- name: Run black
run: make check-style
run-tests:
runs-on: ubuntu-latest
name: Run tests with tox
strategy:
matrix:
python-version: [ '3.6', '3.7', '3.8']
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- uses: Gr1N/setup-poetry@v2
- uses: actions/cache@v2
with:
path: ~/.cache/pypoetry/virtualenvs
key: ${{ runner.os }}-poetry-${{ hashFiles('poetry.lock') }}
restore-keys: |
${{ runner.os }}-poetry-
- name: Install dependencies
run: |
sudo apt install protobuf-compiler libprotobuf-dev
pipenv install --dev
poetry install
- name: Run tests
run: |
cp .env.default .env
pipenv run pip install -e .
pipenv run generate
pipenv run test
- name: Build package
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
run: pipenv run python setup.py sdist
- name: Publish package
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@v1.0.0a0
make generate
make test
build-release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
user: __token__
password: ${{ secrets.pypi }}
python-version: 3.7
- uses: Gr1N/setup-poetry@v2
- name: Build package
run: poetry build
- name: Publish package to PyPI
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
run: poetry publish -n
env:
POETRY_PYPI_TOKEN_PYPI: ${{ secrets.pypi }}

13
.gitignore vendored
View File

@@ -1,13 +1,16 @@
.coverage
.DS_Store
.env
.vscode/settings.json
.mypy_cache
.pytest_cache
betterproto/tests/*.bin
betterproto/tests/*_pb2.py
betterproto/tests/*.py
!betterproto/tests/generate.py
!betterproto/tests/test_*.py
.python-version
build/
betterproto/tests/output_*
**/__pycache__
dist
**/*.egg-info
output
.idea
.DS_Store
.tox

View File

@@ -5,7 +5,26 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [1.2.5] - 2020-04-27
- Add .j2 suffix to python template names to avoid confusing certain build tools [#72](https://github.com/danielgtaylor/python-betterproto/pull/72)
## [1.2.4] - 2020-04-26
- Enforce utf-8 for reading the readme in setup.py [#67](https://github.com/danielgtaylor/python-betterproto/pull/67)
- Only import types from grpclib when type checking [#52](https://github.com/danielgtaylor/python-betterproto/pull/52)
- Improve performance of serialize/deserialize by caching type information of fields in class [#46](https://github.com/danielgtaylor/python-betterproto/pull/46)
- Support using Google's wrapper types as RPC output values [#40](https://github.com/danielgtaylor/python-betterproto/pull/40)
- Fixes issue where protoc did not recognize plugin.py as win32 application [#38](https://github.com/danielgtaylor/python-betterproto/pull/38)
- Fix services using non-pythonified field names [#34](https://github.com/danielgtaylor/python-betterproto/pull/34)
- Add ability to provide metadata, timeout & deadline args to requests [#32](https://github.com/danielgtaylor/python-betterproto/pull/32)
## [1.2.3] - 2020-04-15
- Exclude empty lists from `to_dict` by default [#16](https://github.com/danielgtaylor/python-betterproto/pull/16)
- Add `include_default_values` parameter for `to_dict` [#12](https://github.com/danielgtaylor/python-betterproto/pull/12)
- Fix class names being prepended with duplicates when using protocol buffers that are nested more than once [#21](https://github.com/danielgtaylor/python-betterproto/pull/21)
- Add support for python 3.6 [#30](https://github.com/danielgtaylor/python-betterproto/pull/30)
## [1.2.2] - 2020-01-09
@@ -39,7 +58,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Initial release
[unreleased]: https://github.com/danielgtaylor/python-betterproto/compare/v1.2.2...HEAD
[1.2.5]: https://github.com/danielgtaylor/python-betterproto/compare/v1.2.4...v1.2.5
[1.2.4]: https://github.com/danielgtaylor/python-betterproto/compare/v1.2.3...v1.2.4
[1.2.3]: https://github.com/danielgtaylor/python-betterproto/compare/v1.2.2...v1.2.3
[1.2.2]: https://github.com/danielgtaylor/python-betterproto/compare/v1.2.1...v1.2.2
[1.2.1]: https://github.com/danielgtaylor/python-betterproto/compare/v1.2.0...v1.2.1
[1.2.0]: https://github.com/danielgtaylor/python-betterproto/compare/v1.1.0...v1.2.0

42
Makefile Normal file
View File

@@ -0,0 +1,42 @@
.PHONY: help setup generate test types format clean plugin full-test check-style
help: ## - Show this help.
@fgrep -h "##" $(MAKEFILE_LIST) | fgrep -v fgrep | sed -e 's/\\$$//' | sed -e 's/##//'
# Dev workflow tasks
generate: ## - Generate test cases (do this once before running test)
poetry run ./betterproto/tests/generate.py
test: ## - Run tests
poetry run pytest --cov betterproto
types: ## - Check types with mypy
poetry run mypy betterproto --ignore-missing-imports
format: ## - Apply black formatting to source code
poetry run black . --exclude tests/output_
clean: ## - Clean out generated files from the workspace
rm -rf .coverage \
.mypy_cache \
.pytest_cache \
dist \
**/__pycache__ \
betterproto/tests/output_*
# Manual testing
# By default write plugin output to a directory called output
o=output
plugin: ## - Execute the protoc plugin, with output write to `output` or the value passed to `-o`
mkdir -p $(o)
protoc --plugin=protoc-gen-custom=betterproto/plugin.py $(i) --custom_out=$(o)
# CI tasks
full-test: generate ## - Run full testing sequence with multiple pythons
poetry run tox
check-style: ## - Check if code style is correct
poetry run black . --check --diff --exclude tests/output_

29
Pipfile
View File

@@ -1,29 +0,0 @@
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
flake8 = "*"
mypy = "*"
isort = "*"
pytest = "*"
rope = "*"
[packages]
protobuf = "*"
jinja2 = "*"
grpclib = "*"
stringcase = "*"
black = "*"
[requires]
python_version = "3.7"
[scripts]
plugin = "protoc --plugin=protoc-gen-custom=betterproto/plugin.py --custom_out=output"
generate = "python betterproto/tests/generate.py"
test = "pytest ./betterproto/tests"
[pipenv]
allow_prereleases = true

396
Pipfile.lock generated
View File

@@ -1,396 +0,0 @@
{
"_meta": {
"hash": {
"sha256": "c7b72ed87dc3d70566c53d7ec8a636c8d4854aa30aa97a9116c0734cd5266f33"
},
"pipfile-spec": 6,
"requires": {
"python_version": "3.7"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.org/simple",
"verify_ssl": true
}
]
},
"default": {
"appdirs": {
"hashes": [
"sha256:9e5896d1372858f8dd3344faf4e5014d21849c756c8d5701f78f8a103b372d92",
"sha256:d8b24664561d0d34ddfaec54636d502d7cea6e29c3eaf68f3df6180863e2166e"
],
"version": "==1.4.3"
},
"attrs": {
"hashes": [
"sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c",
"sha256:f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72"
],
"version": "==19.3.0"
},
"black": {
"hashes": [
"sha256:09a9dcb7c46ed496a9850b76e4e825d6049ecd38b611f1224857a79bd985a8cf",
"sha256:68950ffd4d9169716bcb8719a56c07a2f4485354fec061cdd5910aa07369731c"
],
"index": "pypi",
"version": "==19.3b0"
},
"click": {
"hashes": [
"sha256:2335065e6395b9e67ca716de5f7526736bfa6ceead690adf616d925bdc622b13",
"sha256:5b94b49521f6456670fdb30cd82a4eca9412788a93fa6dd6df72c94d5a8ff2d7"
],
"version": "==7.0"
},
"grpclib": {
"hashes": [
"sha256:2d63cee35f764e40a7ea196f27354d2f4ab936401c40b14128bbb4fec06f51d4"
],
"index": "pypi",
"version": "==0.3.1rc2"
},
"h2": {
"hashes": [
"sha256:ac377fcf586314ef3177bfd90c12c7826ab0840edeb03f0f24f511858326049e",
"sha256:b8a32bd282594424c0ac55845377eea13fa54fe4a8db012f3a198ed923dc3ab4"
],
"version": "==3.1.1"
},
"hpack": {
"hashes": [
"sha256:0edd79eda27a53ba5be2dfabf3b15780928a0dff6eb0c60a3d6767720e970c89",
"sha256:8eec9c1f4bfae3408a3f30500261f7e6a65912dc138526ea054f9ad98892e9d2"
],
"version": "==3.0.0"
},
"hyperframe": {
"hashes": [
"sha256:5187962cb16dcc078f23cb5a4b110098d546c3f41ff2d4038a9896893bbd0b40",
"sha256:a9f5c17f2cc3c719b917c4f33ed1c61bd1f8dfac4b1bd23b7c80b3400971b41f"
],
"version": "==5.2.0"
},
"jinja2": {
"hashes": [
"sha256:74320bb91f31270f9551d46522e33af46a80c3d619f4a4bf42b3164d30b5911f",
"sha256:9fe95f19286cfefaa917656583d020be14e7859c6b0252588391e47db34527de"
],
"index": "pypi",
"version": "==2.10.3"
},
"markupsafe": {
"hashes": [
"sha256:00bc623926325b26bb9605ae9eae8a215691f33cae5df11ca5424f06f2d1f473",
"sha256:09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161",
"sha256:09c4b7f37d6c648cb13f9230d847adf22f8171b1ccc4d5682398e77f40309235",
"sha256:1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5",
"sha256:24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff",
"sha256:29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b",
"sha256:43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1",
"sha256:46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e",
"sha256:500d4957e52ddc3351cabf489e79c91c17f6e0899158447047588650b5e69183",
"sha256:535f6fc4d397c1563d08b88e485c3496cf5784e927af890fb3c3aac7f933ec66",
"sha256:62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1",
"sha256:6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1",
"sha256:717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e",
"sha256:79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b",
"sha256:7c1699dfe0cf8ff607dbdcc1e9b9af1755371f92a68f706051cc8c37d447c905",
"sha256:88e5fcfb52ee7b911e8bb6d6aa2fd21fbecc674eadd44118a9cc3863f938e735",
"sha256:8defac2f2ccd6805ebf65f5eeb132adcf2ab57aa11fdf4c0dd5169a004710e7d",
"sha256:98c7086708b163d425c67c7a91bad6e466bb99d797aa64f965e9d25c12111a5e",
"sha256:9add70b36c5666a2ed02b43b335fe19002ee5235efd4b8a89bfcf9005bebac0d",
"sha256:9bf40443012702a1d2070043cb6291650a0841ece432556f784f004937f0f32c",
"sha256:ade5e387d2ad0d7ebf59146cc00c8044acbd863725f887353a10df825fc8ae21",
"sha256:b00c1de48212e4cc9603895652c5c410df699856a2853135b3967591e4beebc2",
"sha256:b1282f8c00509d99fef04d8ba936b156d419be841854fe901d8ae224c59f0be5",
"sha256:b2051432115498d3562c084a49bba65d97cf251f5a331c64a12ee7e04dacc51b",
"sha256:ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6",
"sha256:c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f",
"sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f",
"sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7"
],
"version": "==1.1.1"
},
"multidict": {
"hashes": [
"sha256:024b8129695a952ebd93373e45b5d341dbb87c17ce49637b34000093f243dd4f",
"sha256:041e9442b11409be5e4fc8b6a97e4bcead758ab1e11768d1e69160bdde18acc3",
"sha256:045b4dd0e5f6121e6f314d81759abd2c257db4634260abcfe0d3f7083c4908ef",
"sha256:047c0a04e382ef8bd74b0de01407e8d8632d7d1b4db6f2561106af812a68741b",
"sha256:068167c2d7bbeebd359665ac4fff756be5ffac9cda02375b5c5a7c4777038e73",
"sha256:148ff60e0fffa2f5fad2eb25aae7bef23d8f3b8bdaf947a65cdbe84a978092bc",
"sha256:1d1c77013a259971a72ddaa83b9f42c80a93ff12df6a4723be99d858fa30bee3",
"sha256:1d48bc124a6b7a55006d97917f695effa9725d05abe8ee78fd60d6588b8344cd",
"sha256:31dfa2fc323097f8ad7acd41aa38d7c614dd1960ac6681745b6da124093dc351",
"sha256:34f82db7f80c49f38b032c5abb605c458bac997a6c3142e0d6c130be6fb2b941",
"sha256:3d5dd8e5998fb4ace04789d1d008e2bb532de501218519d70bb672c4c5a2fc5d",
"sha256:4a6ae52bd3ee41ee0f3acf4c60ceb3f44e0e3bc52ab7da1c2b2aa6703363a3d1",
"sha256:4b02a3b2a2f01d0490dd39321c74273fed0568568ea0e7ea23e02bd1fb10a10b",
"sha256:4b843f8e1dd6a3195679d9838eb4670222e8b8d01bc36c9894d6c3538316fa0a",
"sha256:5de53a28f40ef3c4fd57aeab6b590c2c663de87a5af76136ced519923d3efbb3",
"sha256:61b2b33ede821b94fa99ce0b09c9ece049c7067a33b279f343adfe35108a4ea7",
"sha256:6a3a9b0f45fd75dc05d8e93dc21b18fc1670135ec9544d1ad4acbcf6b86781d0",
"sha256:76ad8e4c69dadbb31bad17c16baee61c0d1a4a73bed2590b741b2e1a46d3edd0",
"sha256:7ba19b777dc00194d1b473180d4ca89a054dd18de27d0ee2e42a103ec9b7d014",
"sha256:7c1b7eab7a49aa96f3db1f716f0113a8a2e93c7375dd3d5d21c4941f1405c9c5",
"sha256:7fc0eee3046041387cbace9314926aa48b681202f8897f8bff3809967a049036",
"sha256:8ccd1c5fff1aa1427100ce188557fc31f1e0a383ad8ec42c559aabd4ff08802d",
"sha256:8e08dd76de80539d613654915a2f5196dbccc67448df291e69a88712ea21e24a",
"sha256:c18498c50c59263841862ea0501da9f2b3659c00db54abfbf823a80787fde8ce",
"sha256:c49db89d602c24928e68c0d510f4fcf8989d77defd01c973d6cbe27e684833b1",
"sha256:ce20044d0317649ddbb4e54dab3c1bcc7483c78c27d3f58ab3d0c7e6bc60d26a",
"sha256:d1071414dd06ca2eafa90c85a079169bfeb0e5f57fd0b45d44c092546fcd6fd9",
"sha256:d3be11ac43ab1a3e979dac80843b42226d5d3cccd3986f2e03152720a4297cd7",
"sha256:db603a1c235d110c860d5f39988ebc8218ee028f07a7cbc056ba6424372ca31b"
],
"version": "==4.5.2"
},
"protobuf": {
"hashes": [
"sha256:125713564d8cfed7610e52444c9769b8dcb0b55e25cc7841f2290ee7bc86636f",
"sha256:1accdb7a47e51503be64d9a57543964ba674edac103215576399d2d0e34eac77",
"sha256:27003d12d4f68e3cbea9eb67427cab3bfddd47ff90670cb367fcd7a3a89b9657",
"sha256:3264f3c431a631b0b31e9db2ae8c927b79fc1a7b1b06b31e8e5bcf2af91fe896",
"sha256:3c5ab0f5c71ca5af27143e60613729e3488bb45f6d3f143dc918a20af8bab0bf",
"sha256:45dcf8758873e3f69feab075e5f3177270739f146255225474ee0b90429adef6",
"sha256:56a77d61a91186cc5676d8e11b36a5feb513873e4ae88d2ee5cf530d52bbcd3b",
"sha256:5984e4947bbcef5bd849d6244aec507d31786f2dd3344139adc1489fb403b300",
"sha256:6b0441da73796dd00821763bb4119674eaf252776beb50ae3883bed179a60b2a",
"sha256:6f6677c5ade94d4fe75a912926d6796d5c71a2a90c2aeefe0d6f211d75c74789",
"sha256:84a825a9418d7196e2acc48f8746cf1ee75877ed2f30433ab92a133f3eaf8fbe",
"sha256:b842c34fe043ccf78b4a6cf1019d7b80113707d68c88842d061fa2b8fb6ddedc",
"sha256:ca33d2f09dae149a1dcf942d2d825ebb06343b77b437198c9e2ef115cf5d5bc1",
"sha256:db83b5c12c0cd30150bb568e6feb2435c49ce4e68fe2d7b903113f0e221e58fe",
"sha256:f50f3b1c5c1c1334ca7ce9cad5992f098f460ffd6388a3cabad10b66c2006b09",
"sha256:f99f127909731cafb841c52f9216e447d3e4afb99b17bebfad327a75aee206de"
],
"index": "pypi",
"version": "==3.10.0"
},
"six": {
"hashes": [
"sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c",
"sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
],
"version": "==1.12.0"
},
"stringcase": {
"hashes": [
"sha256:48a06980661908efe8d9d34eab2b6c13aefa2163b3ced26972902e3bdfd87008"
],
"index": "pypi",
"version": "==1.2.0"
},
"toml": {
"hashes": [
"sha256:229f81c57791a41d65e399fc06bf0848bab550a9dfd5ed66df18ce5f05e73d5c",
"sha256:235682dd292d5899d361a811df37e04a8828a5b1da3115886b73cf81ebc9100e"
],
"version": "==0.10.0"
}
},
"develop": {
"atomicwrites": {
"hashes": [
"sha256:03472c30eb2c5d1ba9227e4c2ca66ab8287fbfbbda3888aa93dc2e28fc6811b4",
"sha256:75a9445bac02d8d058d5e1fe689654ba5a6556a1dfd8ce6ec55a0ed79866cfa6"
],
"version": "==1.3.0"
},
"attrs": {
"hashes": [
"sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c",
"sha256:f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72"
],
"version": "==19.3.0"
},
"entrypoints": {
"hashes": [
"sha256:589f874b313739ad35be6e0cd7efde2a4e9b6fea91edcc34e58ecbb8dbe56d19",
"sha256:c70dd71abe5a8c85e55e12c19bd91ccfeec11a6e99044204511f9ed547d48451"
],
"version": "==0.3"
},
"flake8": {
"hashes": [
"sha256:45681a117ecc81e870cbf1262835ae4af5e7a8b08e40b944a8a6e6b895914cfb",
"sha256:49356e766643ad15072a789a20915d3c91dc89fd313ccd71802303fd67e4deca"
],
"index": "pypi",
"version": "==3.7.9"
},
"importlib-metadata": {
"hashes": [
"sha256:aa18d7378b00b40847790e7c27e11673d7fed219354109d0e7b9e5b25dc3ad26",
"sha256:d5f18a79777f3aa179c145737780282e27b508fc8fd688cb17c7a813e8bd39af"
],
"markers": "python_version < '3.8'",
"version": "==0.23"
},
"isort": {
"hashes": [
"sha256:54da7e92468955c4fceacd0c86bd0ec997b0e1ee80d97f67c35a78b719dccab1",
"sha256:6e811fcb295968434526407adb8796944f1988c5b65e8139058f2014cbe100fd"
],
"index": "pypi",
"version": "==4.3.21"
},
"mccabe": {
"hashes": [
"sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
"sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"
],
"version": "==0.6.1"
},
"more-itertools": {
"hashes": [
"sha256:409cd48d4db7052af495b09dec721011634af3753ae1ef92d2b32f73a745f832",
"sha256:92b8c4b06dac4f0611c0729b2f2ede52b2e1bac1ab48f089c7ddc12e26bb60c4"
],
"version": "==7.2.0"
},
"mypy": {
"hashes": [
"sha256:1521c186a3d200c399bd5573c828ea2db1362af7209b2adb1bb8532cea2fb36f",
"sha256:31a046ab040a84a0fc38bc93694876398e62bc9f35eca8ccbf6418b7297f4c00",
"sha256:3b1a411909c84b2ae9b8283b58b48541654b918e8513c20a400bb946aa9111ae",
"sha256:48c8bc99380575deb39f5d3400ebb6a8a1cb5cc669bbba4d3bb30f904e0a0e7d",
"sha256:540c9caa57a22d0d5d3c69047cc9dd0094d49782603eb03069821b41f9e970e9",
"sha256:672e418425d957e276c291930a3921b4a6413204f53fe7c37cad7bc57b9a3391",
"sha256:6ed3b9b3fdc7193ea7aca6f3c20549b377a56f28769783a8f27191903a54170f",
"sha256:9371290aa2cad5ad133e4cdc43892778efd13293406f7340b9ffe99d5ec7c1d9",
"sha256:ace6ac1d0f87d4072f05b5468a084a45b4eda970e4d26704f201e06d47ab2990",
"sha256:b428f883d2b3fe1d052c630642cc6afddd07d5cd7873da948644508be3b9d4a7",
"sha256:d5bf0e6ec8ba346a2cf35cb55bf4adfddbc6b6576fcc9e10863daa523e418dbb",
"sha256:d7574e283f83c08501607586b3167728c58e8442947e027d2d4c7dcd6d82f453",
"sha256:dc889c84241a857c263a2b1cd1121507db7d5b5f5e87e77147097230f374d10b",
"sha256:f4748697b349f373002656bf32fede706a0e713d67bfdcf04edf39b1f61d46eb"
],
"index": "pypi",
"version": "==0.740"
},
"mypy-extensions": {
"hashes": [
"sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d",
"sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"
],
"version": "==0.4.3"
},
"packaging": {
"hashes": [
"sha256:28b924174df7a2fa32c1953825ff29c61e2f5e082343165438812f00d3a7fc47",
"sha256:d9551545c6d761f3def1677baf08ab2a3ca17c56879e70fecba2fc4dde4ed108"
],
"version": "==19.2"
},
"pluggy": {
"hashes": [
"sha256:0db4b7601aae1d35b4a033282da476845aa19185c1e6964b25cf324b5e4ec3e6",
"sha256:fa5fa1622fa6dd5c030e9cad086fa19ef6a0cf6d7a2d12318e10cb49d6d68f34"
],
"version": "==0.13.0"
},
"py": {
"hashes": [
"sha256:64f65755aee5b381cea27766a3a147c3f15b9b6b9ac88676de66ba2ae36793fa",
"sha256:dc639b046a6e2cff5bbe40194ad65936d6ba360b52b3c3fe1d08a82dd50b5e53"
],
"version": "==1.8.0"
},
"pycodestyle": {
"hashes": [
"sha256:95a2219d12372f05704562a14ec30bc76b05a5b297b21a5dfe3f6fac3491ae56",
"sha256:e40a936c9a450ad81df37f549d676d127b1b66000a6c500caa2b085bc0ca976c"
],
"version": "==2.5.0"
},
"pyflakes": {
"hashes": [
"sha256:17dbeb2e3f4d772725c777fabc446d5634d1038f234e77343108ce445ea69ce0",
"sha256:d976835886f8c5b31d47970ed689944a0262b5f3afa00a5a7b4dc81e5449f8a2"
],
"version": "==2.1.1"
},
"pyparsing": {
"hashes": [
"sha256:6f98a7b9397e206d78cc01df10131398f1c8b8510a2f4d97d9abd82e1aacdd80",
"sha256:d9338df12903bbf5d65a0e4e87c2161968b10d2e489652bb47001d82a9b028b4"
],
"version": "==2.4.2"
},
"pytest": {
"hashes": [
"sha256:27abc3fef618a01bebb1f0d6d303d2816a99aa87a5968ebc32fe971be91eb1e6",
"sha256:58cee9e09242937e136dbb3dab466116ba20d6b7828c7620f23947f37eb4dae4"
],
"index": "pypi",
"version": "==5.2.2"
},
"rope": {
"hashes": [
"sha256:6b728fdc3e98a83446c27a91fc5d56808a004f8beab7a31ab1d7224cecc7d969",
"sha256:c5c5a6a87f7b1a2095fb311135e2a3d1f194f5ecb96900fdd0a9100881f48aaf",
"sha256:f0dcf719b63200d492b85535ebe5ea9b29e0d0b8aebeb87fe03fc1a65924fdaf"
],
"index": "pypi",
"version": "==0.14.0"
},
"six": {
"hashes": [
"sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c",
"sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
],
"version": "==1.12.0"
},
"typed-ast": {
"hashes": [
"sha256:1170afa46a3799e18b4c977777ce137bb53c7485379d9706af8a59f2ea1aa161",
"sha256:18511a0b3e7922276346bcb47e2ef9f38fb90fd31cb9223eed42c85d1312344e",
"sha256:262c247a82d005e43b5b7f69aff746370538e176131c32dda9cb0f324d27141e",
"sha256:2b907eb046d049bcd9892e3076c7a6456c93a25bebfe554e931620c90e6a25b0",
"sha256:354c16e5babd09f5cb0ee000d54cfa38401d8b8891eefa878ac772f827181a3c",
"sha256:48e5b1e71f25cfdef98b013263a88d7145879fbb2d5185f2a0c79fa7ebbeae47",
"sha256:4e0b70c6fc4d010f8107726af5fd37921b666f5b31d9331f0bd24ad9a088e631",
"sha256:630968c5cdee51a11c05a30453f8cd65e0cc1d2ad0d9192819df9978984529f4",
"sha256:66480f95b8167c9c5c5c87f32cf437d585937970f3fc24386f313a4c97b44e34",
"sha256:71211d26ffd12d63a83e079ff258ac9d56a1376a25bc80b1cdcdf601b855b90b",
"sha256:7954560051331d003b4e2b3eb822d9dd2e376fa4f6d98fee32f452f52dd6ebb2",
"sha256:838997f4310012cf2e1ad3803bce2f3402e9ffb71ded61b5ee22617b3a7f6b6e",
"sha256:95bd11af7eafc16e829af2d3df510cecfd4387f6453355188342c3e79a2ec87a",
"sha256:bc6c7d3fa1325a0c6613512a093bc2a2a15aeec350451cbdf9e1d4bffe3e3233",
"sha256:cc34a6f5b426748a507dd5d1de4c1978f2eb5626d51326e43280941206c209e1",
"sha256:d755f03c1e4a51e9b24d899561fec4ccaf51f210d52abdf8c07ee2849b212a36",
"sha256:d7c45933b1bdfaf9f36c579671fec15d25b06c8398f113dab64c18ed1adda01d",
"sha256:d896919306dd0aa22d0132f62a1b78d11aaf4c9fc5b3410d3c666b818191630a",
"sha256:fdc1c9bbf79510b76408840e009ed65958feba92a88833cdceecff93ae8fff66",
"sha256:ffde2fbfad571af120fcbfbbc61c72469e72f550d676c3342492a9dfdefb8f12"
],
"version": "==1.4.0"
},
"typing-extensions": {
"hashes": [
"sha256:091ecc894d5e908ac75209f10d5b4f118fbdb2eb1ede6a63544054bb1edb41f2",
"sha256:910f4656f54de5993ad9304959ce9bb903f90aadc7c67a0bef07e678014e892d",
"sha256:cf8b63fedea4d89bab840ecbb93e75578af28f76f66c35889bd7065f5af88575"
],
"version": "==3.7.4.1"
},
"wcwidth": {
"hashes": [
"sha256:3df37372226d6e63e1b1e1eda15c594bca98a22d33a23832a90998faa96bc65e",
"sha256:f4ebe71925af7b40a864553f761ed559b43544f8f71746c2d756c7fe788ade7c"
],
"version": "==0.1.7"
},
"zipp": {
"hashes": [
"sha256:3718b1cbcd963c7d4c5511a8240812904164b7f381b647143a89d3b98f9bcd8e",
"sha256:f06903e9f1f43b12d371004b4ac7b06ab39a44adc747266928ae6debfa7b3335"
],
"version": "==0.6.0"
}
}
}

118
README.md
View File

@@ -6,7 +6,7 @@ This project aims to provide an improved experience when using Protobuf / gRPC i
- Protobuf 3 & gRPC code generation
- Both binary & JSON serialization is built-in
- Python 3.7+ making use of:
- Python 3.6+ making use of:
- Enums
- Dataclasses
- `async`/`await`
@@ -46,10 +46,10 @@ First, install the package. Note that the `[compiler]` feature flag tells it to
```sh
# Install both the library and compiler
$ pip install betterproto[compiler]
pip install "betterproto[compiler]"
# Install just the library (to use the generated code output)
$ pip install betterproto
pip install betterproto
```
Now, given you installed the compiler and have a proto file, e.g `example.proto`:
@@ -68,14 +68,15 @@ message Greeting {
You can run the following:
```sh
$ protoc -I . --python_betterproto_out=. example.proto
mkdir lib
protoc -I . --python_betterproto_out=lib example.proto
```
This will generate `hello.py` which looks like:
This will generate `lib/hello/__init__.py` which looks like:
```py
```python
# Generated by the protocol buffer compiler. DO NOT EDIT!
# sources: hello.proto
# sources: example.proto
# plugin: python-betterproto
from dataclasses import dataclass
@@ -83,7 +84,7 @@ import betterproto
@dataclass
class Hello(betterproto.Message):
class Greeting(betterproto.Message):
"""Greeting represents a message you can tell a user."""
message: str = betterproto.string_field(1)
@@ -91,23 +92,23 @@ class Hello(betterproto.Message):
Now you can use it!
```py
>>> from hello import Hello
>>> test = Hello()
```python
>>> from lib.hello import Greeting
>>> test = Greeting()
>>> test
Hello(message='')
Greeting(message='')
>>> test.message = "Hey!"
>>> test
Hello(message="Hey!")
Greeting(message="Hey!")
>>> serialized = bytes(test)
>>> serialized
b'\n\x04Hey!'
>>> another = Hello().parse(serialized)
>>> another = Greeting().parse(serialized)
>>> another
Hello(message="Hey!")
Greeting(message="Hey!")
>>> another.to_dict()
{"message": "Hey!"}
@@ -256,6 +257,7 @@ Google provides several well-known message types like a timestamp, duration, and
| `google.protobuf.duration` | [`datetime.timedelta`][td] | `0` |
| `google.protobuf.timestamp` | Timezone-aware [`datetime.datetime`][dt] | `1970-01-01T00:00:00Z` |
| `google.protobuf.*Value` | `Optional[...]` | `None` |
| `google.protobuf.*` | `betterproto.lib.google.protobuf.*` | `None` |
[td]: https://docs.python.org/3/library/datetime.html#timedelta-objects
[dt]: https://docs.python.org/3/library/datetime.html#datetime.datetime
@@ -296,36 +298,91 @@ datetime.datetime(2019, 1, 1, 11, 59, 58, 800000, tzinfo=datetime.timezone.utc)
## Development
First, make sure you have Python 3.7+ and `pipenv` installed, along with the official [Protobuf Compiler](https://github.com/protocolbuffers/protobuf/releases) for your platform. Then:
Join us on [Slack](https://join.slack.com/t/betterproto/shared_invite/zt-f0n0uolx-iN8gBNrkPxtKHTLpG3o1OQ)!
First, make sure you have Python 3.6+ and `poetry` installed, along with the official [Protobuf Compiler](https://github.com/protocolbuffers/protobuf/releases) for your platform. Then:
```sh
# Get set up with the virtual env & dependencies
$ pipenv install --dev
poetry install
# Link the local package
$ pipenv shell
$ pip install -e .
# Activate the poetry environment
poetry shell
```
To benefit from the collection of standard development tasks ensure you have make installed and run `make help` to see available tasks.
### Code style
This project enforces [black](https://github.com/psf/black) python code formatting.
Before committing changes run:
```sh
make format
```
To avoid merge conflicts later, non-black formatted python code will fail in CI.
### Tests
There are two types of tests:
1. Manually-written tests for some behavior of the library
2. Proto files and JSON inputs for automated tests
1. Standard tests
2. Custom tests
For #2, you can add a new `*.proto` file into the `betterproto/tests` directory along with a sample `*.json` input and it will get automatically picked up.
#### Standard tests
Adding a standard test case is easy.
- Create a new directory `betterproto/tests/inputs/<name>`
- add `<name>.proto` with a message called `Test`
- add `<name>.json` with some test data (optional)
It will be picked up automatically when you run the tests.
- See also: [Standard Tests Development Guide](betterproto/tests/README.md)
#### Custom tests
Custom tests are found in `tests/test_*.py` and are run with pytest.
#### Running
Here's how to run the tests.
```sh
# Generate assets from sample .proto files
$ pipenv run generate
# Generate assets from sample .proto files required by the tests
make generate
# Run the tests
$ pipenv run tests
make test
```
To run tests as they are run in CI (with tox) run:
```sh
make full-test
```
### (Re)compiling Google Well-known Types
Betterproto includes compiled versions for Google's well-known types at [betterproto/lib/google](betterproto/lib/google).
Be sure to regenerate these files when modifying the plugin output format, and validate by running the tests.
Normally, the plugin does not compile any references to `google.protobuf`, since they are pre-compiled. To force compilation of `google.protobuf`, use the option `--custom_opt=INCLUDE_GOOGLE`.
Assuming your `google.protobuf` source files (included with all releases of `protoc`) are located in `/usr/local/include`, you can regenerate them as follows:
```sh
protoc \
--plugin=protoc-gen-custom=betterproto/plugin.py \
--custom_opt=INCLUDE_GOOGLE \
--custom_out=betterproto/lib \
-I /usr/local/include/ \
/usr/local/include/google/protobuf/*.proto
```
### TODO
- [x] Fixed length fields
@@ -340,6 +397,9 @@ $ pipenv run tests
- [x] Refs to nested types
- [x] Imports in proto files
- [x] Well-known Google types
- [ ] Support as request input
- [ ] Support as response output
- [ ] Automatically wrap/unwrap responses
- [x] OneOf support
- [x] Basic support on the wire
- [x] Check which was set from the group
@@ -363,6 +423,10 @@ $ pipenv run tests
- [x] Automate running tests
- [ ] Cleanup!
## Community
Join us on [Slack](https://join.slack.com/t/betterproto/shared_invite/zt-f0n0uolx-iN8gBNrkPxtKHTLpG3o1OQ)!
## License
Copyright © 2019 Daniel G. Taylor

View File

@@ -3,35 +3,34 @@ import enum
import inspect
import json
import struct
import sys
from abc import ABC
from base64 import b64encode, b64decode
from base64 import b64decode, b64encode
from datetime import datetime, timedelta, timezone
from typing import (
Any,
AsyncGenerator,
Callable,
Dict,
Generator,
Iterable,
List,
Optional,
SupportsBytes,
Set,
Tuple,
Type,
TypeVar,
Union,
get_type_hints,
TYPE_CHECKING,
)
import grpclib.client
import grpclib.const
import stringcase
from ._types import T
from .casing import camel_case, safe_snake_case, safe_snake_case, snake_case
from .grpc.grpclib_client import ServiceStub
from .casing import safe_snake_case
if not (sys.version_info.major == 3 and sys.version_info.minor >= 7):
# Apply backport of datetime.fromisoformat from 3.7
from backports.datetime_fromisoformat import MonkeyPatch
MonkeyPatch.patch_fromisoformat()
if TYPE_CHECKING:
from grpclib._protocols import IProtoMessage
# Proto 3 data types
TYPE_ENUM = "enum"
@@ -110,14 +109,18 @@ WIRE_LEN_DELIM_TYPES = [TYPE_STRING, TYPE_BYTES, TYPE_MESSAGE, TYPE_MAP]
# Protobuf datetimes start at the Unix Epoch in 1970 in UTC.
DATETIME_ZERO = datetime(1970, 1, 1, tzinfo=timezone.utc)
def datetime_default_gen():
return datetime(1970, 1, 1, tzinfo=timezone.utc)
DATETIME_ZERO = datetime_default_gen()
class Casing(enum.Enum):
"""Casing constants for serialization."""
CAMEL = stringcase.camelcase
SNAKE = stringcase.snakecase
CAMEL = camel_case
SNAKE = snake_case
class _PLACEHOLDER:
@@ -414,8 +417,82 @@ def parse_fields(value: bytes) -> Generator[ParsedField, None, None]:
)
# Bound type variable to allow methods to return `self` of subclasses
T = TypeVar("T", bound="Message")
class ProtoClassMetadata:
oneof_group_by_field: Dict[str, str]
oneof_field_by_group: Dict[str, Set[dataclasses.Field]]
default_gen: Dict[str, Callable]
cls_by_field: Dict[str, Type]
field_name_by_number: Dict[int, str]
meta_by_field_name: Dict[str, FieldMetadata]
__slots__ = (
"oneof_group_by_field",
"oneof_field_by_group",
"default_gen",
"cls_by_field",
"field_name_by_number",
"meta_by_field_name",
)
def __init__(self, cls: Type["Message"]):
by_field = {}
by_group: Dict[str, Set] = {}
by_field_name = {}
by_field_number = {}
fields = dataclasses.fields(cls)
for field in fields:
meta = FieldMetadata.get(field)
if meta.group:
# This is part of a one-of group.
by_field[field.name] = meta.group
by_group.setdefault(meta.group, set()).add(field)
by_field_name[field.name] = meta
by_field_number[meta.number] = field.name
self.oneof_group_by_field = by_field
self.oneof_field_by_group = by_group
self.field_name_by_number = by_field_number
self.meta_by_field_name = by_field_name
self.default_gen = self._get_default_gen(cls, fields)
self.cls_by_field = self._get_cls_by_field(cls, fields)
@staticmethod
def _get_default_gen(cls, fields):
default_gen = {}
for field in fields:
default_gen[field.name] = cls._get_field_default_gen(field)
return default_gen
@staticmethod
def _get_cls_by_field(cls, fields):
field_cls = {}
for field in fields:
meta = FieldMetadata.get(field)
if meta.proto_type == TYPE_MAP:
assert meta.map_types
kt = cls._cls_for(field, index=0)
vt = cls._cls_for(field, index=1)
Entry = dataclasses.make_dataclass(
"Entry",
[
("key", kt, dataclass_field(1, meta.map_types[0])),
("value", vt, dataclass_field(2, meta.map_types[1])),
],
bases=(Message,),
)
field_cls[field.name] = Entry
field_cls[field.name + ".value"] = vt
else:
field_cls[field.name] = cls._cls_for(field)
return field_cls
class Message(ABC):
@@ -427,69 +504,74 @@ class Message(ABC):
_serialized_on_wire: bool
_unknown_fields: bytes
_group_map: Dict[str, dict]
_group_current: Dict[str, str]
def __post_init__(self) -> None:
# Keep track of whether every field was default
all_sentinel = True
# Set a default value for each field in the class after `__init__` has
# already been run.
group_map: Dict[str, dict] = {"fields": {}, "groups": {}}
for field in dataclasses.fields(self):
meta = FieldMetadata.get(field)
# Set current field of each group after `__init__` has already been run.
group_current: Dict[str, str] = {}
for field_name, meta in self._betterproto.meta_by_field_name.items():
if meta.group:
# This is part of a one-of group.
group_map["fields"][field.name] = meta.group
group_current.setdefault(meta.group)
if meta.group not in group_map["groups"]:
group_map["groups"][meta.group] = {"current": None, "fields": set()}
group_map["groups"][meta.group]["fields"].add(field)
if getattr(self, field.name) != PLACEHOLDER:
if getattr(self, field_name) != PLACEHOLDER:
# Skip anything not set to the sentinel value
all_sentinel = False
if meta.group:
# This was set, so make it the selected value of the one-of.
group_map["groups"][meta.group]["current"] = field
group_current[meta.group] = field_name
continue
setattr(self, field.name, self._get_field_default(field, meta))
setattr(self, field_name, self._get_field_default(field_name))
# Now that all the defaults are set, reset it!
self.__dict__["_serialized_on_wire"] = not all_sentinel
self.__dict__["_unknown_fields"] = b""
self.__dict__["_group_map"] = group_map
self.__dict__["_group_current"] = group_current
def __setattr__(self, attr: str, value: Any) -> None:
if attr != "_serialized_on_wire":
# Track when a field has been set.
self.__dict__["_serialized_on_wire"] = True
if attr in getattr(self, "_group_map", {}).get("fields", {}):
group = self._group_map["fields"][attr]
for field in self._group_map["groups"][group]["fields"]:
if field.name == attr:
self._group_map["groups"][group]["current"] = field
else:
super().__setattr__(
field.name,
self._get_field_default(field, FieldMetadata.get(field)),
)
if hasattr(self, "_group_current"): # __post_init__ had already run
if attr in self._betterproto.oneof_group_by_field:
group = self._betterproto.oneof_group_by_field[attr]
for field in self._betterproto.oneof_field_by_group[group]:
if field.name == attr:
self._group_current[group] = field.name
else:
super().__setattr__(
field.name, self._get_field_default(field.name),
)
super().__setattr__(attr, value)
@property
def _betterproto(self):
"""
Lazy initialize metadata for each protobuf class.
It may be initialized multiple times in a multi-threaded environment,
but that won't affect the correctness.
"""
meta = getattr(self.__class__, "_betterproto_meta", None)
if not meta:
meta = ProtoClassMetadata(self.__class__)
self.__class__._betterproto_meta = meta
return meta
def __bytes__(self) -> bytes:
"""
Get the binary encoded Protobuf representation of this instance.
"""
output = b""
for field in dataclasses.fields(self):
meta = FieldMetadata.get(field)
value = getattr(self, field.name)
for field_name, meta in self._betterproto.meta_by_field_name.items():
value = getattr(self, field_name)
if value is None:
# Optional items should be skipped. This is used for the Google
@@ -500,16 +582,16 @@ class Message(ABC):
# currently set in a `oneof` group, so it must be serialized even
# if the value is the default zero value.
selected_in_group = False
if meta.group and self._group_map["groups"][meta.group]["current"] == field:
if meta.group and self._group_current[meta.group] == field_name:
selected_in_group = True
serialize_empty = False
if isinstance(value, Message) and value._serialized_on_wire:
# Empty messages can still be sent on the wire if they were
# set (or received empty).
# set (or recieved empty).
serialize_empty = True
if value == self._get_field_default(field, meta) and not (
if value == self._get_field_default(field_name) and not (
selected_in_group or serialize_empty
):
# Default (zero) values are not serialized. Two exceptions are
@@ -552,50 +634,53 @@ class Message(ABC):
# For compatibility with other libraries
SerializeToString = __bytes__
def _type_hint(self, field_name: str) -> Type:
module = inspect.getmodule(self.__class__)
type_hints = get_type_hints(self.__class__, vars(module))
@classmethod
def _type_hint(cls, field_name: str) -> Type:
module = inspect.getmodule(cls)
type_hints = get_type_hints(cls, vars(module))
return type_hints[field_name]
def _cls_for(self, field: dataclasses.Field, index: int = 0) -> Type:
@classmethod
def _cls_for(cls, field: dataclasses.Field, index: int = 0) -> Type:
"""Get the message class for a field from the type hints."""
cls = self._type_hint(field.name)
if hasattr(cls, "__args__") and index >= 0:
cls = cls.__args__[index]
return cls
field_cls = cls._type_hint(field.name)
if hasattr(field_cls, "__args__") and index >= 0:
field_cls = field_cls.__args__[index]
return field_cls
def _get_field_default(self, field: dataclasses.Field, meta: FieldMetadata) -> Any:
t = self._type_hint(field.name)
def _get_field_default(self, field_name):
return self._betterproto.default_gen[field_name]()
@classmethod
def _get_field_default_gen(cls, field: dataclasses.Field) -> Any:
t = cls._type_hint(field.name)
value: Any = 0
if hasattr(t, "__origin__"):
if t.__origin__ == dict:
if t.__origin__ in (dict, Dict):
# This is some kind of map (dict in Python).
value = {}
elif t.__origin__ == list:
return dict
elif t.__origin__ in (list, List):
# This is some kind of list (repeated) field.
value = []
return list
elif t.__origin__ == Union and t.__args__[1] == type(None):
# This is an optional (wrapped) field. For setting the default we
# really don't care what kind of field it is.
value = None
return type(None)
else:
value = t()
return t
elif issubclass(t, Enum):
# Enums always default to zero.
value = 0
return int
elif t == datetime:
# Offsets are relative to 1970-01-01T00:00:00Z
value = DATETIME_ZERO
return datetime_default_gen
else:
# This is either a primitive scalar or another message type. Calling
# it should result in its zero value.
value = t()
return value
return t
def _postprocess_single(
self, wire_type: int, meta: FieldMetadata, field: dataclasses.Field, value: Any
self, wire_type: int, meta: FieldMetadata, field_name: str, value: Any
) -> Any:
"""Adjusts values after parsing."""
if wire_type == WIRE_VARINT:
@@ -617,7 +702,7 @@ class Message(ABC):
if meta.proto_type == TYPE_STRING:
value = value.decode("utf-8")
elif meta.proto_type == TYPE_MESSAGE:
cls = self._cls_for(field)
cls = self._betterproto.cls_by_field[field_name]
if cls == datetime:
value = _Timestamp().parse(value).to_datetime()
@@ -631,20 +716,7 @@ class Message(ABC):
value = cls().parse(value)
value._serialized_on_wire = True
elif meta.proto_type == TYPE_MAP:
# TODO: This is slow, use a cache to make it faster since each
# key/value pair will recreate the class.
assert meta.map_types
kt = self._cls_for(field, index=0)
vt = self._cls_for(field, index=1)
Entry = dataclasses.make_dataclass(
"Entry",
[
("key", kt, dataclass_field(1, meta.map_types[0])),
("value", vt, dataclass_field(2, meta.map_types[1])),
],
bases=(Message,),
)
value = Entry().parse(value)
value = self._betterproto.cls_by_field[field_name]().parse(value)
return value
@@ -653,49 +725,46 @@ class Message(ABC):
Parse the binary encoded Protobuf into this message instance. This
returns the instance itself and is therefore assignable and chainable.
"""
fields = {f.metadata["betterproto"].number: f for f in dataclasses.fields(self)}
for parsed in parse_fields(data):
if parsed.number in fields:
field = fields[parsed.number]
meta = FieldMetadata.get(field)
value: Any
if (
parsed.wire_type == WIRE_LEN_DELIM
and meta.proto_type in PACKED_TYPES
):
# This is a packed repeated field.
pos = 0
value = []
while pos < len(parsed.value):
if meta.proto_type in ["float", "fixed32", "sfixed32"]:
decoded, pos = parsed.value[pos : pos + 4], pos + 4
wire_type = WIRE_FIXED_32
elif meta.proto_type in ["double", "fixed64", "sfixed64"]:
decoded, pos = parsed.value[pos : pos + 8], pos + 8
wire_type = WIRE_FIXED_64
else:
decoded, pos = decode_varint(parsed.value, pos)
wire_type = WIRE_VARINT
decoded = self._postprocess_single(
wire_type, meta, field, decoded
)
value.append(decoded)
else:
value = self._postprocess_single(
parsed.wire_type, meta, field, parsed.value
)
current = getattr(self, field.name)
if meta.proto_type == TYPE_MAP:
# Value represents a single key/value pair entry in the map.
current[value.key] = value.value
elif isinstance(current, list) and not isinstance(value, list):
current.append(value)
else:
setattr(self, field.name, value)
else:
field_name = self._betterproto.field_name_by_number.get(parsed.number)
if not field_name:
self._unknown_fields += parsed.raw
continue
meta = self._betterproto.meta_by_field_name[field_name]
value: Any
if parsed.wire_type == WIRE_LEN_DELIM and meta.proto_type in PACKED_TYPES:
# This is a packed repeated field.
pos = 0
value = []
while pos < len(parsed.value):
if meta.proto_type in ["float", "fixed32", "sfixed32"]:
decoded, pos = parsed.value[pos : pos + 4], pos + 4
wire_type = WIRE_FIXED_32
elif meta.proto_type in ["double", "fixed64", "sfixed64"]:
decoded, pos = parsed.value[pos : pos + 8], pos + 8
wire_type = WIRE_FIXED_64
else:
decoded, pos = decode_varint(parsed.value, pos)
wire_type = WIRE_VARINT
decoded = self._postprocess_single(
wire_type, meta, field_name, decoded
)
value.append(decoded)
else:
value = self._postprocess_single(
parsed.wire_type, meta, field_name, parsed.value
)
current = getattr(self, field_name)
if meta.proto_type == TYPE_MAP:
# Value represents a single key/value pair entry in the map.
current[value.key] = value.value
elif isinstance(current, list) and not isinstance(value, list):
current.append(value)
else:
setattr(self, field_name, value)
return self
@@ -704,41 +773,49 @@ class Message(ABC):
def FromString(cls: Type[T], data: bytes) -> T:
return cls().parse(data)
def to_dict(self, casing: Casing = Casing.CAMEL) -> dict:
def to_dict(
self, casing: Casing = Casing.CAMEL, include_default_values: bool = False
) -> Dict[str, Any]:
"""
Returns a dict representation of this message instance which can be
used to serialize to e.g. JSON. Defaults to camel casing for
compatibility but can be set to other modes.
`include_default_values` can be set to `True` to include default
values of fields. E.g. an `int32` type field with `0` value will
not be in returned dict if `include_default_values` is set to
`False`.
"""
output: Dict[str, Any] = {}
for field in dataclasses.fields(self):
meta = FieldMetadata.get(field)
v = getattr(self, field.name)
cased_name = casing(field.name).rstrip("_") # type: ignore
for field_name, meta in self._betterproto.meta_by_field_name.items():
v = getattr(self, field_name)
cased_name = casing(field_name).rstrip("_") # type: ignore
if meta.proto_type == "message":
if isinstance(v, datetime):
if v != DATETIME_ZERO:
if v != DATETIME_ZERO or include_default_values:
output[cased_name] = _Timestamp.timestamp_to_json(v)
elif isinstance(v, timedelta):
if v != timedelta(0):
if v != timedelta(0) or include_default_values:
output[cased_name] = _Duration.delta_to_json(v)
elif meta.wraps:
if v is not None:
if v is not None or include_default_values:
output[cased_name] = v
elif isinstance(v, list):
# Convert each item.
v = [i.to_dict(casing) for i in v]
output[cased_name] = v
elif v._serialized_on_wire:
output[cased_name] = v.to_dict(casing)
v = [i.to_dict(casing, include_default_values) for i in v]
if v or include_default_values:
output[cased_name] = v
else:
if v._serialized_on_wire or include_default_values:
output[cased_name] = v.to_dict(casing, include_default_values)
elif meta.proto_type == "map":
for k in v:
if hasattr(v[k], "to_dict"):
v[k] = v[k].to_dict(casing)
v[k] = v[k].to_dict(casing, include_default_values)
if v:
if v or include_default_values:
output[cased_name] = v
elif v != self._get_field_default(field, meta):
elif v != self._get_field_default(field_name) or include_default_values:
if meta.proto_type in INT_64_TYPES:
if isinstance(v, list):
output[cased_name] = [str(n) for n in v]
@@ -750,7 +827,9 @@ class Message(ABC):
else:
output[cased_name] = b64encode(v).decode("utf8")
elif meta.proto_type == TYPE_ENUM:
enum_values = list(self._cls_for(field)) # type: ignore
enum_values = list(
self._betterproto.cls_by_field[field_name]
) # type: ignore
if isinstance(v, list):
output[cased_name] = [enum_values[e].name for e in v]
else:
@@ -767,56 +846,54 @@ class Message(ABC):
self._serialized_on_wire = True
fields_by_name = {f.name: f for f in dataclasses.fields(self)}
for key in value:
snake_cased = safe_snake_case(key)
if snake_cased in fields_by_name:
field = fields_by_name[snake_cased]
meta = FieldMetadata.get(field)
field_name = safe_snake_case(key)
meta = self._betterproto.meta_by_field_name.get(field_name)
if not meta:
continue
if value[key] is not None:
if meta.proto_type == "message":
v = getattr(self, field.name)
if isinstance(v, list):
cls = self._cls_for(field)
for i in range(len(value[key])):
v.append(cls().from_dict(value[key][i]))
elif isinstance(v, datetime):
v = datetime.fromisoformat(
value[key].replace("Z", "+00:00")
)
setattr(self, field.name, v)
elif isinstance(v, timedelta):
v = timedelta(seconds=float(value[key][:-1]))
setattr(self, field.name, v)
elif meta.wraps:
setattr(self, field.name, value[key])
else:
v.from_dict(value[key])
elif meta.map_types and meta.map_types[1] == TYPE_MESSAGE:
v = getattr(self, field.name)
cls = self._cls_for(field, index=1)
for k in value[key]:
v[k] = cls().from_dict(value[key][k])
if value[key] is not None:
if meta.proto_type == "message":
v = getattr(self, field_name)
if isinstance(v, list):
cls = self._betterproto.cls_by_field[field_name]
for i in range(len(value[key])):
v.append(cls().from_dict(value[key][i]))
elif isinstance(v, datetime):
v = datetime.fromisoformat(value[key].replace("Z", "+00:00"))
setattr(self, field_name, v)
elif isinstance(v, timedelta):
v = timedelta(seconds=float(value[key][:-1]))
setattr(self, field_name, v)
elif meta.wraps:
setattr(self, field_name, value[key])
else:
v = value[key]
if meta.proto_type in INT_64_TYPES:
if isinstance(value[key], list):
v = [int(n) for n in value[key]]
else:
v = int(value[key])
elif meta.proto_type == TYPE_BYTES:
if isinstance(value[key], list):
v = [b64decode(n) for n in value[key]]
else:
v = b64decode(value[key])
elif meta.proto_type == TYPE_ENUM:
enum_cls = self._cls_for(field)
if isinstance(v, list):
v = [enum_cls.from_string(e) for e in v]
elif isinstance(v, str):
v = enum_cls.from_string(v)
v.from_dict(value[key])
elif meta.map_types and meta.map_types[1] == TYPE_MESSAGE:
v = getattr(self, field_name)
cls = self._betterproto.cls_by_field[field_name + ".value"]
for k in value[key]:
v[k] = cls().from_dict(value[key][k])
else:
v = value[key]
if meta.proto_type in INT_64_TYPES:
if isinstance(value[key], list):
v = [int(n) for n in value[key]]
else:
v = int(value[key])
elif meta.proto_type == TYPE_BYTES:
if isinstance(value[key], list):
v = [b64decode(n) for n in value[key]]
else:
v = b64decode(value[key])
elif meta.proto_type == TYPE_ENUM:
enum_cls = self._betterproto.cls_by_field[field_name]
if isinstance(v, list):
v = [enum_cls.from_string(e) for e in v]
elif isinstance(v, str):
v = enum_cls.from_string(v)
if v is not None:
setattr(self, field.name, v)
if v is not None:
setattr(self, field_name, v)
return self
def to_json(self, indent: Union[None, int, str] = None) -> str:
@@ -842,25 +919,29 @@ def serialized_on_wire(message: Message) -> bool:
def which_one_of(message: Message, group_name: str) -> Tuple[str, Any]:
"""Return the name and value of a message's one-of field group."""
field = message._group_map["groups"].get(group_name, {}).get("current")
if not field:
field_name = message._group_current.get(group_name)
if not field_name:
return ("", None)
return (field.name, getattr(message, field.name))
return (field_name, getattr(message, field_name))
@dataclasses.dataclass
class _Duration(Message):
# Signed seconds of the span of time. Must be from -315,576,000,000 to
# +315,576,000,000 inclusive. Note: these bounds are computed from: 60
# sec/min * 60 min/hr * 24 hr/day * 365.25 days/year * 10000 years
seconds: int = int64_field(1)
# Signed fractions of a second at nanosecond resolution of the span of time.
# Durations less than one second are represented with a 0 `seconds` field and
# a positive or negative `nanos` field. For durations of one second or more,
# a non-zero value for the `nanos` field must be of the same sign as the
# `seconds` field. Must be from -999,999,999 to +999,999,999 inclusive.
nanos: int = int32_field(2)
# Circular import workaround: google.protobuf depends on base classes defined above.
from .lib.google.protobuf import (
Duration,
Timestamp,
BoolValue,
BytesValue,
DoubleValue,
FloatValue,
Int32Value,
Int64Value,
StringValue,
UInt32Value,
UInt64Value,
)
class _Duration(Duration):
def to_timedelta(self) -> timedelta:
return timedelta(seconds=self.seconds, microseconds=self.nanos / 1e3)
@@ -873,16 +954,7 @@ class _Duration(Message):
return ".".join(parts) + "s"
@dataclasses.dataclass
class _Timestamp(Message):
# Represents seconds of UTC time since Unix epoch 1970-01-01T00:00:00Z. Must
# be from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59Z inclusive.
seconds: int = int64_field(1)
# Non-negative fractions of a second at nanosecond resolution. Negative
# second values with fractions must still have non-negative nanos values that
# count forward in time. Must be from 0 to 999,999,999 inclusive.
nanos: int = int32_field(2)
class _Timestamp(Timestamp):
def to_datetime(self) -> datetime:
ts = self.seconds + (self.nanos / 1e9)
return datetime.fromtimestamp(ts, tz=timezone.utc)
@@ -923,93 +995,16 @@ class _WrappedMessage(Message):
return self
@dataclasses.dataclass
class _BoolValue(_WrappedMessage):
value: bool = bool_field(1)
@dataclasses.dataclass
class _Int32Value(_WrappedMessage):
value: int = int32_field(1)
@dataclasses.dataclass
class _UInt32Value(_WrappedMessage):
value: int = uint32_field(1)
@dataclasses.dataclass
class _Int64Value(_WrappedMessage):
value: int = int64_field(1)
@dataclasses.dataclass
class _UInt64Value(_WrappedMessage):
value: int = uint64_field(1)
@dataclasses.dataclass
class _FloatValue(_WrappedMessage):
value: float = float_field(1)
@dataclasses.dataclass
class _DoubleValue(_WrappedMessage):
value: float = double_field(1)
@dataclasses.dataclass
class _StringValue(_WrappedMessage):
value: str = string_field(1)
@dataclasses.dataclass
class _BytesValue(_WrappedMessage):
value: bytes = bytes_field(1)
def _get_wrapper(proto_type: str) -> Type:
"""Get the wrapper message class for a wrapped type."""
return {
TYPE_BOOL: _BoolValue,
TYPE_INT32: _Int32Value,
TYPE_UINT32: _UInt32Value,
TYPE_INT64: _Int64Value,
TYPE_UINT64: _UInt64Value,
TYPE_FLOAT: _FloatValue,
TYPE_DOUBLE: _DoubleValue,
TYPE_STRING: _StringValue,
TYPE_BYTES: _BytesValue,
TYPE_BOOL: BoolValue,
TYPE_INT32: Int32Value,
TYPE_UINT32: UInt32Value,
TYPE_INT64: Int64Value,
TYPE_UINT64: UInt64Value,
TYPE_FLOAT: FloatValue,
TYPE_DOUBLE: DoubleValue,
TYPE_STRING: StringValue,
TYPE_BYTES: BytesValue,
}[proto_type]
class ServiceStub(ABC):
"""
Base class for async gRPC service stubs.
"""
def __init__(self, channel: grpclib.client.Channel) -> None:
self.channel = channel
async def _unary_unary(
self, route: str, request: "IProtoMessage", response_type: Type[T]
) -> T:
"""Make a unary request and return the response."""
async with self.channel.request(
route, grpclib.const.Cardinality.UNARY_UNARY, type(request), response_type
) as stream:
await stream.send_message(request, end=True)
response = await stream.recv_message()
assert response is not None
return response
async def _unary_stream(
self, route: str, request: "IProtoMessage", response_type: Type[T]
) -> AsyncGenerator[T, None]:
"""Make a unary request and return the stream response iterator."""
async with self.channel.request(
route, grpclib.const.Cardinality.UNARY_STREAM, type(request), response_type
) as stream:
await stream.send_message(request, end=True)
async for message in stream:
yield message

9
betterproto/_types.py Normal file
View File

@@ -0,0 +1,9 @@
from typing import TYPE_CHECKING, TypeVar
if TYPE_CHECKING:
from . import Message
from grpclib._protocols import IProtoMessage
# Bound type variable to allow methods to return `self` of subclasses
T = TypeVar("T", bound="Message")
ST = TypeVar("ST", bound="IProtoMessage")

View File

@@ -1,9 +1,21 @@
import stringcase
import re
# Word delimiters and symbols that will not be preserved when re-casing.
# language=PythonRegExp
SYMBOLS = "[^a-zA-Z0-9]*"
# Optionally capitalized word.
# language=PythonRegExp
WORD = "[A-Z]*[a-z]*[0-9]*"
# Uppercase word, not followed by lowercase letters.
# language=PythonRegExp
WORD_UPPER = "[A-Z]+(?![a-z])[0-9]*"
def safe_snake_case(value: str) -> str:
"""Snake case a value taking into account Python keywords."""
value = stringcase.snakecase(value)
value = snake_case(value)
if value in [
"and",
"as",
@@ -39,3 +51,70 @@ def safe_snake_case(value: str) -> str:
# https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles
value += "_"
return value
def snake_case(value: str, strict: bool = True):
"""
Join words with an underscore into lowercase and remove symbols.
@param value: value to convert
@param strict: force single underscores
"""
def substitute_word(symbols, word, is_start):
if not word:
return ""
if strict:
delimiter_count = 0 if is_start else 1 # Single underscore if strict.
elif is_start:
delimiter_count = len(symbols)
elif word.isupper() or word.islower():
delimiter_count = max(
1, len(symbols)
) # Preserve all delimiters if not strict.
else:
delimiter_count = len(symbols) + 1 # Extra underscore for leading capital.
return ("_" * delimiter_count) + word.lower()
snake = re.sub(
f"(^)?({SYMBOLS})({WORD_UPPER}|{WORD})",
lambda groups: substitute_word(groups[2], groups[3], groups[1] is not None),
value,
)
return snake
def pascal_case(value: str, strict: bool = True):
"""
Capitalize each word and remove symbols.
@param value: value to convert
@param strict: output only alphanumeric characters
"""
def substitute_word(symbols, word):
if strict:
return word.capitalize() # Remove all delimiters
if word.islower():
delimiter_length = len(symbols[:-1]) # Lose one delimiter
else:
delimiter_length = len(symbols) # Preserve all delimiters
return ("_" * delimiter_length) + word.capitalize()
return re.sub(
f"({SYMBOLS})({WORD_UPPER}|{WORD})",
lambda groups: substitute_word(groups[1], groups[2]),
value,
)
def camel_case(value: str, strict: bool = True):
"""
Capitalize all words except first and remove symbols.
"""
return lowercase_first(pascal_case(value, strict=strict))
def lowercase_first(value: str):
return value[0:1].lower() + value[1:]

View File

View File

@@ -0,0 +1,160 @@
import os
import re
from typing import Dict, List, Set, Type
from betterproto import safe_snake_case
from betterproto.compile.naming import pythonize_class_name
from betterproto.lib.google import protobuf as google_protobuf
WRAPPER_TYPES: Dict[str, Type] = {
".google.protobuf.DoubleValue": google_protobuf.DoubleValue,
".google.protobuf.FloatValue": google_protobuf.FloatValue,
".google.protobuf.Int32Value": google_protobuf.Int32Value,
".google.protobuf.Int64Value": google_protobuf.Int64Value,
".google.protobuf.UInt32Value": google_protobuf.UInt32Value,
".google.protobuf.UInt64Value": google_protobuf.UInt64Value,
".google.protobuf.BoolValue": google_protobuf.BoolValue,
".google.protobuf.StringValue": google_protobuf.StringValue,
".google.protobuf.BytesValue": google_protobuf.BytesValue,
}
def parse_source_type_name(field_type_name):
"""
Split full source type name into package and type name.
E.g. 'root.package.Message' -> ('root.package', 'Message')
'root.Message.SomeEnum' -> ('root', 'Message.SomeEnum')
"""
package_match = re.match(r"^\.?([^A-Z]+)\.(.+)", field_type_name)
if package_match:
package = package_match.group(1)
name = package_match.group(2)
else:
package = ""
name = field_type_name.lstrip(".")
return package, name
def get_type_reference(
package: str, imports: set, source_type: str, unwrap: bool = True,
) -> str:
"""
Return a Python type name for a proto type reference. Adds the import if
necessary. Unwraps well known type if required.
"""
if unwrap:
if source_type in WRAPPER_TYPES:
wrapped_type = type(WRAPPER_TYPES[source_type]().value)
return f"Optional[{wrapped_type.__name__}]"
if source_type == ".google.protobuf.Duration":
return "timedelta"
if source_type == ".google.protobuf.Timestamp":
return "datetime"
source_package, source_type = parse_source_type_name(source_type)
current_package: List[str] = package.split(".") if package else []
py_package: List[str] = source_package.split(".") if source_package else []
py_type: str = pythonize_class_name(source_type)
compiling_google_protobuf = current_package == ["google", "protobuf"]
importing_google_protobuf = py_package == ["google", "protobuf"]
if importing_google_protobuf and not compiling_google_protobuf:
py_package = ["betterproto", "lib"] + py_package
if py_package[:1] == ["betterproto"]:
return reference_absolute(imports, py_package, py_type)
if py_package == current_package:
return reference_sibling(py_type)
if py_package[: len(current_package)] == current_package:
return reference_descendent(current_package, imports, py_package, py_type)
if current_package[: len(py_package)] == py_package:
return reference_ancestor(current_package, imports, py_package, py_type)
return reference_cousin(current_package, imports, py_package, py_type)
def reference_absolute(imports, py_package, py_type):
"""
Returns a reference to a python type located in the root, i.e. sys.path.
"""
string_import = ".".join(py_package)
string_alias = safe_snake_case(string_import)
imports.add(f"import {string_import} as {string_alias}")
return f"{string_alias}.{py_type}"
def reference_sibling(py_type: str) -> str:
"""
Returns a reference to a python type within the same package as the current package.
"""
return f'"{py_type}"'
def reference_descendent(
current_package: List[str], imports: Set[str], py_package: List[str], py_type: str
) -> str:
"""
Returns a reference to a python type in a package that is a descendent of the current package,
and adds the required import that is aliased to avoid name conflicts.
"""
importing_descendent = py_package[len(current_package) :]
string_from = ".".join(importing_descendent[:-1])
string_import = importing_descendent[-1]
if string_from:
string_alias = "_".join(importing_descendent)
imports.add(f"from .{string_from} import {string_import} as {string_alias}")
return f"{string_alias}.{py_type}"
else:
imports.add(f"from . import {string_import}")
return f"{string_import}.{py_type}"
def reference_ancestor(
current_package: List[str], imports: Set[str], py_package: List[str], py_type: str
) -> str:
"""
Returns a reference to a python type in a package which is an ancestor to the current package,
and adds the required import that is aliased (if possible) to avoid name conflicts.
Adds trailing __ to avoid name mangling (python.org/dev/peps/pep-0008/#id34).
"""
distance_up = len(current_package) - len(py_package)
if py_package:
string_import = py_package[-1]
string_alias = f"_{'_' * distance_up}{string_import}__"
string_from = f"..{'.' * distance_up}"
imports.add(f"from {string_from} import {string_import} as {string_alias}")
return f"{string_alias}.{py_type}"
else:
string_alias = f"{'_' * distance_up}{py_type}__"
imports.add(f"from .{'.' * distance_up} import {py_type} as {string_alias}")
return string_alias
def reference_cousin(
current_package: List[str], imports: Set[str], py_package: List[str], py_type: str
) -> str:
"""
Returns a reference to a python type in a package that is not descendent, ancestor or sibling,
and adds the required import that is aliased to avoid name conflicts.
"""
shared_ancestry = os.path.commonprefix([current_package, py_package])
distance_up = len(current_package) - len(shared_ancestry)
string_from = f".{'.' * distance_up}" + ".".join(
py_package[len(shared_ancestry) : -1]
)
string_import = py_package[-1]
# Add trailing __ to avoid name mangling (python.org/dev/peps/pep-0008/#id34)
string_alias = (
f"{'_' * distance_up}"
+ safe_snake_case(".".join(py_package[len(shared_ancestry) :]))
+ "__"
)
imports.add(f"from {string_from} import {string_import} as {string_alias}")
return f"{string_alias}.{py_type}"

View File

@@ -0,0 +1,13 @@
from betterproto import casing
def pythonize_class_name(name):
return casing.pascal_case(name)
def pythonize_field_name(name: str):
return casing.safe_snake_case(name)
def pythonize_method_name(name: str):
return casing.safe_snake_case(name)

View File

View File

@@ -0,0 +1,170 @@
from abc import ABC
import asyncio
import grpclib.const
from typing import (
Any,
AsyncIterable,
AsyncIterator,
Collection,
Iterable,
Mapping,
Optional,
Tuple,
TYPE_CHECKING,
Type,
Union,
)
from .._types import ST, T
if TYPE_CHECKING:
from grpclib._protocols import IProtoMessage
from grpclib.client import Channel, Stream
from grpclib.metadata import Deadline
_Value = Union[str, bytes]
_MetadataLike = Union[Mapping[str, _Value], Collection[Tuple[str, _Value]]]
_MessageSource = Union[Iterable["IProtoMessage"], AsyncIterable["IProtoMessage"]]
class ServiceStub(ABC):
"""
Base class for async gRPC clients.
"""
def __init__(
self,
channel: "Channel",
*,
timeout: Optional[float] = None,
deadline: Optional["Deadline"] = None,
metadata: Optional[_MetadataLike] = None,
) -> None:
self.channel = channel
self.timeout = timeout
self.deadline = deadline
self.metadata = metadata
def __resolve_request_kwargs(
self,
timeout: Optional[float],
deadline: Optional["Deadline"],
metadata: Optional[_MetadataLike],
):
return {
"timeout": self.timeout if timeout is None else timeout,
"deadline": self.deadline if deadline is None else deadline,
"metadata": self.metadata if metadata is None else metadata,
}
async def _unary_unary(
self,
route: str,
request: "IProtoMessage",
response_type: Type[T],
*,
timeout: Optional[float] = None,
deadline: Optional["Deadline"] = None,
metadata: Optional[_MetadataLike] = None,
) -> T:
"""Make a unary request and return the response."""
async with self.channel.request(
route,
grpclib.const.Cardinality.UNARY_UNARY,
type(request),
response_type,
**self.__resolve_request_kwargs(timeout, deadline, metadata),
) as stream:
await stream.send_message(request, end=True)
response = await stream.recv_message()
assert response is not None
return response
async def _unary_stream(
self,
route: str,
request: "IProtoMessage",
response_type: Type[T],
*,
timeout: Optional[float] = None,
deadline: Optional["Deadline"] = None,
metadata: Optional[_MetadataLike] = None,
) -> AsyncIterator[T]:
"""Make a unary request and return the stream response iterator."""
async with self.channel.request(
route,
grpclib.const.Cardinality.UNARY_STREAM,
type(request),
response_type,
**self.__resolve_request_kwargs(timeout, deadline, metadata),
) as stream:
await stream.send_message(request, end=True)
async for message in stream:
yield message
async def _stream_unary(
self,
route: str,
request_iterator: _MessageSource,
request_type: Type[ST],
response_type: Type[T],
*,
timeout: Optional[float] = None,
deadline: Optional["Deadline"] = None,
metadata: Optional[_MetadataLike] = None,
) -> T:
"""Make a stream request and return the response."""
async with self.channel.request(
route,
grpclib.const.Cardinality.STREAM_UNARY,
request_type,
response_type,
**self.__resolve_request_kwargs(timeout, deadline, metadata),
) as stream:
await self._send_messages(stream, request_iterator)
response = await stream.recv_message()
assert response is not None
return response
async def _stream_stream(
self,
route: str,
request_iterator: _MessageSource,
request_type: Type[ST],
response_type: Type[T],
*,
timeout: Optional[float] = None,
deadline: Optional["Deadline"] = None,
metadata: Optional[_MetadataLike] = None,
) -> AsyncIterator[T]:
"""
Make a stream request and return an AsyncIterator to iterate over response
messages.
"""
async with self.channel.request(
route,
grpclib.const.Cardinality.STREAM_STREAM,
request_type,
response_type,
**self.__resolve_request_kwargs(timeout, deadline, metadata),
) as stream:
await stream.send_request()
sending_task = asyncio.ensure_future(
self._send_messages(stream, request_iterator)
)
try:
async for response in stream:
yield response
except:
sending_task.cancel()
raise
@staticmethod
async def _send_messages(stream, messages: _MessageSource):
if isinstance(messages, AsyncIterable):
async for message in messages:
await stream.send_message(message)
else:
for message in messages:
await stream.send_message(message)
await stream.end()

View File

View File

@@ -0,0 +1,198 @@
import asyncio
from typing import (
AsyncIterable,
AsyncIterator,
Iterable,
Optional,
TypeVar,
Union,
)
T = TypeVar("T")
class ChannelClosed(Exception):
"""
An exception raised on an attempt to send through a closed channel
"""
pass
class ChannelDone(Exception):
"""
An exception raised on an attempt to send recieve from a channel that is both closed
and empty.
"""
pass
class AsyncChannel(AsyncIterable[T]):
"""
A buffered async channel for sending items between coroutines with FIFO ordering.
This makes decoupled bidirection steaming gRPC requests easy if used like:
.. code-block:: python
client = GeneratedStub(grpclib_chan)
request_chan = await AsyncChannel()
# We can start be sending all the requests we already have
await request_chan.send_from([ReqestObject(...), ReqestObject(...)])
async for response in client.rpc_call(request_chan):
# The response iterator will remain active until the connection is closed
...
# More items can be sent at any time
await request_chan.send(ReqestObject(...))
...
# The channel must be closed to complete the gRPC connection
request_chan.close()
Items can be sent through the channel by either:
- providing an iterable to the send_from method
- passing them to the send method one at a time
Items can be recieved from the channel by either:
- iterating over the channel with a for loop to get all items
- calling the recieve method to get one item at a time
If the channel is empty then recievers will wait until either an item appears or the
channel is closed.
Once the channel is closed then subsequent attempt to send through the channel will
fail with a ChannelClosed exception.
When th channel is closed and empty then it is done, and further attempts to recieve
from it will fail with a ChannelDone exception
If multiple coroutines recieve from the channel concurrently, each item sent will be
recieved by only one of the recievers.
:param source:
An optional iterable will items that should be sent through the channel
immediately.
:param buffer_limit:
Limit the number of items that can be buffered in the channel, A value less than
1 implies no limit. If the channel is full then attempts to send more items will
result in the sender waiting until an item is recieved from the channel.
:param close:
If set to True then the channel will automatically close after exhausting source
or immediately if no source is provided.
"""
def __init__(
self, *, buffer_limit: int = 0, close: bool = False,
):
self._queue: asyncio.Queue[Union[T, object]] = asyncio.Queue(buffer_limit)
self._closed = False
self._waiting_recievers: int = 0
# Track whether flush has been invoked so it can only happen once
self._flushed = False
def __aiter__(self) -> AsyncIterator[T]:
return self
async def __anext__(self) -> T:
if self.done():
raise StopAsyncIteration
self._waiting_recievers += 1
try:
result = await self._queue.get()
if result is self.__flush:
raise StopAsyncIteration
return result
finally:
self._waiting_recievers -= 1
self._queue.task_done()
def closed(self) -> bool:
"""
Returns True if this channel is closed and no-longer accepting new items
"""
return self._closed
def done(self) -> bool:
"""
Check if this channel is done.
:return: True if this channel is closed and and has been drained of items in
which case any further attempts to recieve an item from this channel will raise
a ChannelDone exception.
"""
# After close the channel is not yet done until there is at least one waiting
# reciever per enqueued item.
return self._closed and self._queue.qsize() <= self._waiting_recievers
async def send_from(
self, source: Union[Iterable[T], AsyncIterable[T]], close: bool = False
) -> "AsyncChannel[T]":
"""
Iterates the given [Async]Iterable and sends all the resulting items.
If close is set to True then subsequent send calls will be rejected with a
ChannelClosed exception.
:param source: an iterable of items to send
:param close:
if True then the channel will be closed after the source has been exhausted
"""
if self._closed:
raise ChannelClosed("Cannot send through a closed channel")
if isinstance(source, AsyncIterable):
async for item in source:
await self._queue.put(item)
else:
for item in source:
await self._queue.put(item)
if close:
# Complete the closing process
self.close()
return self
async def send(self, item: T) -> "AsyncChannel[T]":
"""
Send a single item over this channel.
:param item: The item to send
"""
if self._closed:
raise ChannelClosed("Cannot send through a closed channel")
await self._queue.put(item)
return self
async def recieve(self) -> Optional[T]:
"""
Returns the next item from this channel when it becomes available,
or None if the channel is closed before another item is sent.
:return: An item from the channel
"""
if self.done():
raise ChannelDone("Cannot recieve from a closed channel")
self._waiting_recievers += 1
try:
result = await self._queue.get()
if result is self.__flush:
return None
return result
finally:
self._waiting_recievers -= 1
self._queue.task_done()
def close(self):
"""
Close this channel to new items
"""
self._closed = True
asyncio.ensure_future(self._flush_queue())
async def _flush_queue(self):
"""
To be called after the channel is closed. Pushes a number of self.__flush
objects to the queue to ensure no waiting consumers get deadlocked.
"""
if not self._flushed:
self._flushed = True
deadlocked_recievers = max(0, self._waiting_recievers - self._queue.qsize())
for _ in range(deadlocked_recievers):
await self._queue.put(self.__flush)
# A special signal object for flushing the queue when the channel is closed
__flush = object()

View File

View File

File diff suppressed because it is too large Load Diff

2
betterproto/plugin.bat Normal file
View File

@@ -0,0 +1,2 @@
@SET plugin_dir=%~dp0
@python %plugin_dir%/plugin.py %*

View File

@@ -1,112 +1,65 @@
#!/usr/bin/env python
import itertools
import json
import os.path
import pathlib
import re
import sys
import textwrap
from typing import Any, List, Tuple
from typing import List, Union
import betterproto
from betterproto.compile.importing import get_type_reference
from betterproto.compile.naming import (
pythonize_class_name,
pythonize_field_name,
pythonize_method_name,
)
try:
# betterproto[compiler] specific dependencies
import black
except ImportError:
from google.protobuf.compiler import plugin_pb2 as plugin
from google.protobuf.descriptor_pb2 import (
DescriptorProto,
EnumDescriptorProto,
FieldDescriptorProto,
)
import google.protobuf.wrappers_pb2 as google_wrappers
import jinja2
except ImportError as err:
missing_import = err.args[0][17:-1]
print(
"Unable to import `black` formatter. Did you install the compiler feature with `pip install betterproto[compiler]`?"
"\033[31m"
f"Unable to import `{missing_import}` from betterproto plugin! "
"Please ensure that you've installed betterproto as "
'`pip install "betterproto[compiler]"` so that compiler dependencies '
"are included."
"\033[0m"
)
raise SystemExit(1)
import jinja2
import stringcase
from google.protobuf.compiler import plugin_pb2 as plugin
from google.protobuf.descriptor_pb2 import (
DescriptorProto,
EnumDescriptorProto,
FieldDescriptorProto,
FileDescriptorProto,
ServiceDescriptorProto,
)
from betterproto.casing import safe_snake_case
WRAPPER_TYPES = {
"google.protobuf.DoubleValue": "float",
"google.protobuf.FloatValue": "float",
"google.protobuf.Int64Value": "int",
"google.protobuf.UInt64Value": "int",
"google.protobuf.Int32Value": "int",
"google.protobuf.UInt32Value": "int",
"google.protobuf.BoolValue": "bool",
"google.protobuf.StringValue": "str",
"google.protobuf.BytesValue": "bytes",
}
def get_ref_type(package: str, imports: set, type_name: str) -> str:
"""
Return a Python type name for a proto type reference. Adds the import if
necessary.
"""
# If the package name is a blank string, then this should still work
# because by convention packages are lowercase and message/enum types are
# pascal-cased. May require refactoring in the future.
type_name = type_name.lstrip(".")
if type_name in WRAPPER_TYPES:
return f"Optional[{WRAPPER_TYPES[type_name]}]"
if type_name == "google.protobuf.Duration":
return "timedelta"
if type_name == "google.protobuf.Timestamp":
return "datetime"
if type_name.startswith(package):
parts = type_name.lstrip(package).lstrip(".").split(".")
if len(parts) == 1 or (len(parts) > 1 and parts[0][0] == parts[0][0].upper()):
# This is the current package, which has nested types flattened.
# foo.bar_thing => FooBarThing
cased = [stringcase.pascalcase(part) for part in parts]
type_name = f'"{"".join(cased)}"'
if "." in type_name:
# This is imported from another package. No need
# to use a forward ref and we need to add the import.
parts = type_name.split(".")
parts[-1] = stringcase.pascalcase(parts[-1])
imports.add(f"from .{'.'.join(parts[:-2])} import {parts[-2]}")
type_name = f"{parts[-2]}.{parts[-1]}"
return type_name
def py_type(
package: str,
imports: set,
message: DescriptorProto,
descriptor: FieldDescriptorProto,
) -> str:
if descriptor.type in [1, 2, 6, 7, 15, 16]:
def py_type(package: str, imports: set, field: FieldDescriptorProto) -> str:
if field.type in [1, 2]:
return "float"
elif descriptor.type in [3, 4, 5, 13, 17, 18]:
elif field.type in [3, 4, 5, 6, 7, 13, 15, 16, 17, 18]:
return "int"
elif descriptor.type == 8:
elif field.type == 8:
return "bool"
elif descriptor.type == 9:
elif field.type == 9:
return "str"
elif descriptor.type in [11, 14]:
elif field.type in [11, 14]:
# Type referencing another defined Message or a named enum
return get_ref_type(package, imports, descriptor.type_name)
elif descriptor.type == 12:
return get_type_reference(package, imports, field.type_name)
elif field.type == 12:
return "bytes"
else:
raise NotImplementedError(f"Unknown type {descriptor.type}")
raise NotImplementedError(f"Unknown type {field.type}")
def get_py_zero(type_num: int) -> str:
zero = 0
def get_py_zero(type_num: int) -> Union[str, float]:
zero: Union[str, float] = 0
if type_num in []:
zero = 0.0
elif type_num == 8:
@@ -122,19 +75,19 @@ def get_py_zero(type_num: int) -> str:
def traverse(proto_file):
def _traverse(path, items):
def _traverse(path, items, prefix=""):
for i, item in enumerate(items):
# Adjust the name since we flatten the heirarchy.
item.name = next_prefix = prefix + item.name
yield item, path + [i]
if isinstance(item, DescriptorProto):
for enum in item.enum_type:
enum.name = item.name + enum.name
enum.name = next_prefix + enum.name
yield enum, path + [i, 4]
if item.nested_type:
for n, p in _traverse(path + [i, 3], item.nested_type):
# Adjust the name since we flatten the heirarchy.
n.name = item.name + n.name
for n, p in _traverse(path + [i, 3], item.nested_type, next_prefix):
yield n, p
return itertools.chain(
@@ -167,25 +120,28 @@ def get_comment(proto_file, path: List[int], indent: int = 4) -> str:
def generate_code(request, response):
plugin_options = request.parameter.split(",") if request.parameter else []
env = jinja2.Environment(
trim_blocks=True,
lstrip_blocks=True,
loader=jinja2.FileSystemLoader("%s/templates/" % os.path.dirname(__file__)),
)
template = env.get_template("template.py")
template = env.get_template("template.py.j2")
output_map = {}
for proto_file in request.proto_file:
out = proto_file.package
if out == "google.protobuf":
if (
proto_file.package == "google.protobuf"
and "INCLUDE_GOOGLE" not in plugin_options
):
continue
if not out:
out = os.path.splitext(proto_file.name)[0].replace(os.path.sep, ".")
output_file = str(pathlib.Path(*proto_file.package.split("."), "__init__.py"))
if out not in output_map:
output_map[out] = {"package": proto_file.package, "files": []}
output_map[out]["files"].append(proto_file)
if output_file not in output_map:
output_map[output_file] = {"package": proto_file.package, "files": []}
output_map[output_file]["files"].append(proto_file)
# TODO: Figure out how to handle gRPC request/response messages and add
# processing below for Service.
@@ -204,17 +160,10 @@ def generate_code(request, response):
"services": [],
}
type_mapping = {}
for proto_file in options["files"]:
# print(proto_file.message_type, file=sys.stderr)
# print(proto_file.service, file=sys.stderr)
# print(proto_file.source_code_info, file=sys.stderr)
item: DescriptorProto
for item, path in traverse(proto_file):
# print(item, file=sys.stderr)
# print(path, file=sys.stderr)
data = {"name": item.name, "py_name": stringcase.pascalcase(item.name)}
data = {"name": item.name, "py_name": pythonize_class_name(item.name)}
if isinstance(item, DescriptorProto):
# print(item, file=sys.stderr)
@@ -231,7 +180,7 @@ def generate_code(request, response):
)
for i, f in enumerate(item.field):
t = py_type(package, output["imports"], item, f)
t = py_type(package, output["imports"], f)
zero = get_py_zero(f.type)
repeated = False
@@ -240,11 +189,13 @@ def generate_code(request, response):
field_type = f.Type.Name(f.type).lower()[5:]
field_wraps = ""
if f.type_name.startswith(
".google.protobuf"
) and f.type_name.endswith("Value"):
w = f.type_name.split(".").pop()[:-5].upper()
field_wraps = f"betterproto.TYPE_{w}"
match_wrapper = re.match(
r"\.google\.protobuf\.(.+)Value", f.type_name
)
if match_wrapper:
wrapped_type = "TYPE_" + match_wrapper.group(1).upper()
if hasattr(betterproto, wrapped_type):
field_wraps = f"betterproto.{wrapped_type}"
map_types = None
if f.type == 11:
@@ -264,13 +215,11 @@ def generate_code(request, response):
k = py_type(
package,
output["imports"],
item,
nested.field[0],
)
v = py_type(
package,
output["imports"],
item,
nested.field[1],
)
t = f"Dict[{k}, {v}]"
@@ -306,7 +255,7 @@ def generate_code(request, response):
data["properties"].append(
{
"name": f.name,
"py_name": safe_snake_case(f.name),
"py_name": pythonize_field_name(f.name),
"number": f.number,
"comment": get_comment(proto_file, path + [2, i]),
"proto_type": int(f.type),
@@ -347,17 +296,14 @@ def generate_code(request, response):
data = {
"name": service.name,
"py_name": stringcase.pascalcase(service.name),
"py_name": pythonize_class_name(service.name),
"comment": get_comment(proto_file, [6, i]),
"methods": [],
}
for j, method in enumerate(service.method):
if method.client_streaming:
raise NotImplementedError("Client streaming not yet supported")
input_message = None
input_type = get_ref_type(
input_type = get_type_reference(
package, output["imports"], method.input_type
).strip('"')
for msg in output["messages"]:
@@ -371,23 +317,30 @@ def generate_code(request, response):
data["methods"].append(
{
"name": method.name,
"py_name": stringcase.snakecase(method.name),
"py_name": pythonize_method_name(method.name),
"comment": get_comment(proto_file, [6, i, 2, j], indent=8),
"route": f"/{package}.{service.name}/{method.name}",
"input": get_ref_type(
"input": get_type_reference(
package, output["imports"], method.input_type
).strip('"'),
"input_message": input_message,
"output": get_ref_type(
package, output["imports"], method.output_type
"output": get_type_reference(
package,
output["imports"],
method.output_type,
unwrap=False,
).strip('"'),
"client_streaming": method.client_streaming,
"server_streaming": method.server_streaming,
}
)
if method.client_streaming:
output["typing_imports"].add("AsyncIterable")
output["typing_imports"].add("Iterable")
output["typing_imports"].add("Union")
if method.server_streaming:
output["typing_imports"].add("AsyncGenerator")
output["typing_imports"].add("AsyncIterator")
output["services"].append(data)
@@ -397,8 +350,7 @@ def generate_code(request, response):
# Fill response
f = response.file.add()
# print(filename, file=sys.stderr)
f.name = filename.replace(".", os.path.sep) + ".py"
f.name = filename
# Render and then format the output file.
f.content = black.format_str(
@@ -406,32 +358,23 @@ def generate_code(request, response):
mode=black.FileMode(target_versions=set([black.TargetVersion.PY37])),
)
inits = set([""])
for f in response.file:
# Ensure output paths exist
# print(f.name, file=sys.stderr)
dirnames = os.path.dirname(f.name)
if dirnames:
os.makedirs(dirnames, exist_ok=True)
base = ""
for part in dirnames.split(os.path.sep):
base = os.path.join(base, part)
inits.add(base)
for base in inits:
name = os.path.join(base, "__init__.py")
if os.path.exists(name):
# Never overwrite inits as they may have custom stuff in them.
continue
# Make each output directory a package with __init__ file
output_paths = set(pathlib.Path(path) for path in output_map.keys())
init_files = (
set(
directory.joinpath("__init__.py")
for path in output_paths
for directory in path.parents
)
- output_paths
)
for init_file in init_files:
init = response.file.add()
init.name = name
init.content = b""
init.name = str(init_file)
filenames = sorted([f.name for f in response.file])
for fname in filenames:
print(f"Writing {fname}", file=sys.stderr)
for filename in sorted(output_paths.union(init_files)):
print(f"Writing {filename}", file=sys.stderr)
def main():

View File

@@ -63,34 +63,72 @@ class {{ service.py_name }}Stub(betterproto.ServiceStub):
{% endif %}
{% for method in service.methods %}
async def {{ method.py_name }}(self{% if method.input_message and method.input_message.properties %}, *, {% for field in method.input_message.properties %}{{ field.name }}: {% if field.zero == "None" and not field.type.startswith("Optional[") %}Optional[{{ field.type }}]{% else %}{{ field.type }}{% endif %} = {{ field.zero }}{% if not loop.last %}, {% endif %}{% endfor %}{% endif %}) -> {% if method.server_streaming %}AsyncGenerator[{{ method.output }}, None]{% else %}{{ method.output }}{% endif %}:
async def {{ method.py_name }}(self
{%- if not method.client_streaming -%}
{%- if method.input_message and method.input_message.properties -%}, *,
{%- for field in method.input_message.properties -%}
{{ field.py_name }}: {% if field.zero == "None" and not field.type.startswith("Optional[") -%}
Optional[{{ field.type }}]
{%- else -%}
{{ field.type }}
{%- endif -%} = {{ field.zero }}
{%- if not loop.last %}, {% endif -%}
{%- endfor -%}
{%- endif -%}
{%- else -%}
{# Client streaming: need a request iterator instead #}
, request_iterator: Union[AsyncIterable["{{ method.input }}"], Iterable["{{ method.input }}"]]
{%- endif -%}
) -> {% if method.server_streaming %}AsyncIterator[{{ method.output }}]{% else %}{{ method.output }}{% endif %}:
{% if method.comment %}
{{ method.comment }}
{% endif %}
{% if not method.client_streaming %}
request = {{ method.input }}()
{% for field in method.input_message.properties %}
{% if field.field_type == 'message' %}
if {{ field.name }} is not None:
request.{{ field.name }} = {{ field.name }}
if {{ field.py_name }} is not None:
request.{{ field.py_name }} = {{ field.py_name }}
{% else %}
request.{{ field.name }} = {{ field.name }}
request.{{ field.py_name }} = {{ field.py_name }}
{% endif %}
{% endfor %}
{% endif %}
{% if method.server_streaming %}
{% if method.client_streaming %}
async for response in self._stream_stream(
"{{ method.route }}",
request_iterator,
{{ method.input }},
{{ method.output }},
):
yield response
{% else %}{# i.e. not client streaming #}
async for response in self._unary_stream(
"{{ method.route }}",
request,
{{ method.output }},
):
yield response
{% else %}
{% endif %}{# if client streaming #}
{% else %}{# i.e. not server streaming #}
{% if method.client_streaming %}
return await self._stream_unary(
"{{ method.route }}",
request_iterator,
{{ method.input }},
{{ method.output }}
)
{% else %}{# i.e. not client streaming #}
return await self._unary_unary(
"{{ method.route }}",
request,
{{ method.output }},
{{ method.output }}
)
{% endif %}{# client streaming #}
{% endif %}
{% endfor %}

View File

@@ -0,0 +1,91 @@
# Standard Tests Development Guide
Standard test cases are found in [betterproto/tests/inputs](inputs), where each subdirectory represents a testcase, that is verified in isolation.
```
inputs/
bool/
double/
int32/
...
```
## Test case directory structure
Each testcase has a `<name>.proto` file with a message called `Test`, and optionally a matching `.json` file and a custom test called `test_*.py`.
```bash
bool/
bool.proto
bool.json # optional
test_bool.py # optional
```
### proto
`<name>.proto` &mdash; *The protobuf message to test*
```protobuf
syntax = "proto3";
message Test {
bool value = 1;
}
```
You can add multiple `.proto` files to the test case, as long as one file matches the directory name.
### json
`<name>.json` &mdash; *Test-data to validate the message with*
```json
{
"value": true
}
```
### pytest
`test_<name>.py` &mdash; *Custom test to validate specific aspects of the generated class*
```python
from betterproto.tests.output_betterproto.bool.bool import Test
def test_value():
message = Test()
assert not message.value, "Boolean is False by default"
```
## Standard tests
The following tests are automatically executed for all cases:
- [x] Can the generated python code be imported?
- [x] Can the generated message class be instantiated?
- [x] Is the generated code compatible with the Google's `grpc_tools.protoc` implementation?
- _when `.json` is present_
## Running the tests
- `pipenv run generate`
This generates:
- `betterproto/tests/output_betterproto` &mdash; *the plugin generated python classes*
- `betterproto/tests/output_reference` &mdash; *reference implementation classes*
- `pipenv run test`
## Intentionally Failing tests
The standard test suite includes tests that fail by intention. These tests document known bugs and missing features that are intended to be corrected in the future.
When running `pytest`, they show up as `x` or `X` in the test results.
```
betterproto/tests/test_inputs.py ..x...x..x...x.X........xx........x.....x.......x.xx....x...................... [ 84%]
```
- `.` &mdash; PASSED
- `x` &mdash; XFAIL: expected failure
- `X` &mdash; XPASS: expected failure, but still passed
Test cases marked for expected failure are declared in [inputs/config.py](inputs/config.py)

View File

189
betterproto/tests/generate.py Normal file → Executable file
View File

@@ -1,84 +1,143 @@
#!/usr/bin/env python
import asyncio
import os
from pathlib import Path
import shutil
import subprocess
import sys
from typing import Set
from betterproto.tests.util import (
get_directories,
inputs_path,
output_path_betterproto,
output_path_reference,
protoc_plugin,
protoc_reference,
)
# Force pure-python implementation instead of C++, otherwise imports
# break things because we can't properly reset the symbol database.
os.environ["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"] = "python"
import importlib
import json
import subprocess
import sys
from typing import Generator, Tuple
from google.protobuf import symbol_database
from google.protobuf.descriptor_pool import DescriptorPool
from google.protobuf.json_format import MessageToJson, Parse
def clear_directory(dir_path: Path):
for file_or_directory in dir_path.glob("*"):
if file_or_directory.is_dir():
shutil.rmtree(file_or_directory)
else:
file_or_directory.unlink()
root = os.path.dirname(os.path.realpath(__file__))
async def generate(whitelist: Set[str], verbose: bool):
test_case_names = set(get_directories(inputs_path)) - {"__pycache__"}
path_whitelist = set()
name_whitelist = set()
for item in whitelist:
if item in test_case_names:
name_whitelist.add(item)
continue
path_whitelist.add(item)
generation_tasks = []
for test_case_name in sorted(test_case_names):
test_case_input_path = inputs_path.joinpath(test_case_name).resolve()
if (
whitelist
and str(test_case_input_path) not in path_whitelist
and test_case_name not in name_whitelist
):
continue
generation_tasks.append(
generate_test_case_output(test_case_input_path, test_case_name, verbose)
)
failed_test_cases = []
# Wait for all subprocs and match any failures to names to report
for test_case_name, result in zip(
sorted(test_case_names), await asyncio.gather(*generation_tasks)
):
if result != 0:
failed_test_cases.append(test_case_name)
if failed_test_cases:
sys.stderr.write(
"\n\033[31;1;4mFailed to generate the following test cases:\033[0m\n"
)
for failed_test_case in failed_test_cases:
sys.stderr.write(f"- {failed_test_case}\n")
def get_files(end: str) -> Generator[str, None, None]:
for r, dirs, files in os.walk(root):
for filename in [f for f in files if f.endswith(end)]:
yield os.path.join(r, filename)
async def generate_test_case_output(
test_case_input_path: Path, test_case_name: str, verbose: bool
) -> int:
"""
Returns the max of the subprocess return values
"""
test_case_output_path_reference = output_path_reference.joinpath(test_case_name)
test_case_output_path_betterproto = output_path_betterproto.joinpath(test_case_name)
os.makedirs(test_case_output_path_reference, exist_ok=True)
os.makedirs(test_case_output_path_betterproto, exist_ok=True)
clear_directory(test_case_output_path_reference)
clear_directory(test_case_output_path_betterproto)
(
(ref_out, ref_err, ref_code),
(plg_out, plg_err, plg_code),
) = await asyncio.gather(
protoc_reference(test_case_input_path, test_case_output_path_reference),
protoc_plugin(test_case_input_path, test_case_output_path_betterproto),
)
message = f"Generated output for {test_case_name!r}"
if verbose:
print(f"\033[31;1;4m{message}\033[0m")
if ref_out:
sys.stdout.buffer.write(ref_out)
if ref_err:
sys.stderr.buffer.write(ref_err)
if plg_out:
sys.stdout.buffer.write(plg_out)
if plg_err:
sys.stderr.buffer.write(plg_err)
sys.stdout.buffer.flush()
sys.stderr.buffer.flush()
else:
print(message)
return max(ref_code, plg_code)
def get_base(filename: str) -> str:
return os.path.splitext(os.path.basename(filename))[0]
HELP = "\n".join(
(
"Usage: python generate.py [-h] [-v] [DIRECTORIES or NAMES]",
"Generate python classes for standard tests.",
"",
"DIRECTORIES One or more relative or absolute directories of test-cases to generate classes for.",
" python generate.py inputs/bool inputs/double inputs/enum",
"",
"NAMES One or more test-case names to generate classes for.",
" python generate.py bool double enums",
)
)
def ensure_ext(filename: str, ext: str) -> str:
if not filename.endswith(ext):
return filename + ext
return filename
def main():
if set(sys.argv).intersection({"-h", "--help"}):
print(HELP)
return
if sys.argv[1:2] == ["-v"]:
verbose = True
whitelist = set(sys.argv[2:])
else:
verbose = False
whitelist = set(sys.argv[1:])
asyncio.get_event_loop().run_until_complete(generate(whitelist, verbose))
if __name__ == "__main__":
os.chdir(root)
if len(sys.argv) > 1:
proto_files = [ensure_ext(f, ".proto") for f in sys.argv[1:]]
bases = {get_base(f) for f in proto_files}
json_files = [
f for f in get_files(".json") if get_base(f).split("-")[0] in bases
]
else:
proto_files = get_files(".proto")
json_files = get_files(".json")
for filename in proto_files:
print(f"Generating code for {os.path.basename(filename)}")
subprocess.run(
f"protoc --python_out=. {os.path.basename(filename)}", shell=True
)
subprocess.run(
f"protoc --plugin=protoc-gen-custom=../plugin.py --custom_out=. {os.path.basename(filename)}",
shell=True,
)
for filename in json_files:
# Reset the internal symbol database so we can import the `Test` message
# multiple times. Ugh.
sym = symbol_database.Default()
sym.pool = DescriptorPool()
parts = get_base(filename).split("-")
out = filename.replace(".json", ".bin")
print(f"Using {parts[0]}_pb2 to generate {os.path.basename(out)}")
imported = importlib.import_module(f"{parts[0]}_pb2")
input_json = open(filename).read()
parsed = Parse(input_json, imported.Test())
serialized = parsed.SerializeToString()
preserve = "casing" not in filename
serialized_json = MessageToJson(parsed, preserving_proto_field_name=preserve)
s_loaded = json.loads(serialized_json)
in_loaded = json.loads(input_json)
if s_loaded != in_loaded:
raise AssertionError("Expected JSON to be equal:", s_loaded, in_loaded)
open(out, "wb").write(serialized)
main()

View File

View File

@@ -0,0 +1,154 @@
import asyncio
from betterproto.tests.output_betterproto.service.service import (
DoThingResponse,
DoThingRequest,
GetThingRequest,
GetThingResponse,
TestStub as ThingServiceClient,
)
import grpclib
from grpclib.testing import ChannelFor
import pytest
from betterproto.grpc.util.async_channel import AsyncChannel
from .thing_service import ThingService
async def _test_client(client, name="clean room", **kwargs):
response = await client.do_thing(name=name)
assert response.names == [name]
def _assert_request_meta_recieved(deadline, metadata):
def server_side_test(stream):
assert stream.deadline._timestamp == pytest.approx(
deadline._timestamp, 1
), "The provided deadline should be recieved serverside"
assert (
stream.metadata["authorization"] == metadata["authorization"]
), "The provided authorization metadata should be recieved serverside"
return server_side_test
@pytest.mark.asyncio
async def test_simple_service_call():
async with ChannelFor([ThingService()]) as channel:
await _test_client(ThingServiceClient(channel))
@pytest.mark.asyncio
async def test_service_call_with_upfront_request_params():
# Setting deadline
deadline = grpclib.metadata.Deadline.from_timeout(22)
metadata = {"authorization": "12345"}
async with ChannelFor(
[ThingService(test_hook=_assert_request_meta_recieved(deadline, metadata),)]
) as channel:
await _test_client(
ThingServiceClient(channel, deadline=deadline, metadata=metadata)
)
# Setting timeout
timeout = 99
deadline = grpclib.metadata.Deadline.from_timeout(timeout)
metadata = {"authorization": "12345"}
async with ChannelFor(
[ThingService(test_hook=_assert_request_meta_recieved(deadline, metadata),)]
) as channel:
await _test_client(
ThingServiceClient(channel, timeout=timeout, metadata=metadata)
)
@pytest.mark.asyncio
async def test_service_call_lower_level_with_overrides():
THING_TO_DO = "get milk"
# Setting deadline
deadline = grpclib.metadata.Deadline.from_timeout(22)
metadata = {"authorization": "12345"}
kwarg_deadline = grpclib.metadata.Deadline.from_timeout(28)
kwarg_metadata = {"authorization": "12345"}
async with ChannelFor(
[ThingService(test_hook=_assert_request_meta_recieved(deadline, metadata),)]
) as channel:
client = ThingServiceClient(channel, deadline=deadline, metadata=metadata)
response = await client._unary_unary(
"/service.Test/DoThing",
DoThingRequest(THING_TO_DO),
DoThingResponse,
deadline=kwarg_deadline,
metadata=kwarg_metadata,
)
assert response.names == [THING_TO_DO]
# Setting timeout
timeout = 99
deadline = grpclib.metadata.Deadline.from_timeout(timeout)
metadata = {"authorization": "12345"}
kwarg_timeout = 9000
kwarg_deadline = grpclib.metadata.Deadline.from_timeout(kwarg_timeout)
kwarg_metadata = {"authorization": "09876"}
async with ChannelFor(
[
ThingService(
test_hook=_assert_request_meta_recieved(kwarg_deadline, kwarg_metadata),
)
]
) as channel:
client = ThingServiceClient(channel, deadline=deadline, metadata=metadata)
response = await client._unary_unary(
"/service.Test/DoThing",
DoThingRequest(THING_TO_DO),
DoThingResponse,
timeout=kwarg_timeout,
metadata=kwarg_metadata,
)
assert response.names == [THING_TO_DO]
@pytest.mark.asyncio
async def test_async_gen_for_unary_stream_request():
thing_name = "my milkshakes"
async with ChannelFor([ThingService()]) as channel:
client = ThingServiceClient(channel)
expected_versions = [5, 4, 3, 2, 1]
async for response in client.get_thing_versions(name=thing_name):
assert response.name == thing_name
assert response.version == expected_versions.pop()
@pytest.mark.asyncio
async def test_async_gen_for_stream_stream_request():
some_things = ["cake", "cricket", "coral reef"]
more_things = ["ball", "that", "56kmodem", "liberal humanism", "cheesesticks"]
expected_things = (*some_things, *more_things)
async with ChannelFor([ThingService()]) as channel:
client = ThingServiceClient(channel)
# Use an AsyncChannel to decouple sending and recieving, it'll send some_things
# immediately and we'll use it to send more_things later, after recieving some
# results
request_chan = AsyncChannel()
send_initial_requests = asyncio.ensure_future(
request_chan.send_from(GetThingRequest(name) for name in some_things)
)
response_index = 0
async for response in client.get_different_things(request_chan):
assert response.name == expected_things[response_index]
assert response.version == response_index + 1
response_index += 1
if more_things:
# Send some more requests as we recieve reponses to be sure coordination of
# send/recieve events doesn't matter
await request_chan.send(GetThingRequest(more_things.pop(0)))
elif not send_initial_requests.done():
# Make sure the sending task it completed
await send_initial_requests
else:
# No more things to send make sure channel is closed
request_chan.close()
assert response_index == len(
expected_things
), "Didn't recieve all exptected responses"

View File

@@ -0,0 +1,100 @@
import asyncio
import betterproto
from betterproto.grpc.util.async_channel import AsyncChannel
from dataclasses import dataclass
import pytest
from typing import AsyncIterator
@dataclass
class Message(betterproto.Message):
body: str = betterproto.string_field(1)
@pytest.fixture
def expected_responses():
return [Message("Hello world 1"), Message("Hello world 2"), Message("Done")]
class ClientStub:
async def connect(self, requests: AsyncIterator):
await asyncio.sleep(0.1)
async for request in requests:
await asyncio.sleep(0.1)
yield request
await asyncio.sleep(0.1)
yield Message("Done")
async def to_list(generator: AsyncIterator):
result = []
async for value in generator:
result.append(value)
return result
@pytest.fixture
def client():
# channel = Channel(host='127.0.0.1', port=50051)
# return ClientStub(channel)
return ClientStub()
@pytest.mark.asyncio
async def test_send_from_before_connect_and_close_automatically(
client, expected_responses
):
requests = AsyncChannel()
await requests.send_from(
[Message(body="Hello world 1"), Message(body="Hello world 2")], close=True
)
responses = client.connect(requests)
assert await to_list(responses) == expected_responses
@pytest.mark.asyncio
async def test_send_from_after_connect_and_close_automatically(
client, expected_responses
):
requests = AsyncChannel()
responses = client.connect(requests)
await requests.send_from(
[Message(body="Hello world 1"), Message(body="Hello world 2")], close=True
)
assert await to_list(responses) == expected_responses
@pytest.mark.asyncio
async def test_send_from_close_manually_immediately(client, expected_responses):
requests = AsyncChannel()
responses = client.connect(requests)
await requests.send_from(
[Message(body="Hello world 1"), Message(body="Hello world 2")], close=False
)
requests.close()
assert await to_list(responses) == expected_responses
@pytest.mark.asyncio
async def test_send_individually_and_close_before_connect(client, expected_responses):
requests = AsyncChannel()
await requests.send(Message(body="Hello world 1"))
await requests.send(Message(body="Hello world 2"))
requests.close()
responses = client.connect(requests)
assert await to_list(responses) == expected_responses
@pytest.mark.asyncio
async def test_send_individually_and_close_after_connect(client, expected_responses):
requests = AsyncChannel()
await requests.send(Message(body="Hello world 1"))
await requests.send(Message(body="Hello world 2"))
responses = client.connect(requests)
requests.close()
assert await to_list(responses) == expected_responses

View File

@@ -0,0 +1,83 @@
from betterproto.tests.output_betterproto.service.service import (
DoThingResponse,
DoThingRequest,
GetThingRequest,
GetThingResponse,
TestStub as ThingServiceClient,
)
import grpclib
from typing import Any, Dict
class ThingService:
def __init__(self, test_hook=None):
# This lets us pass assertions to the servicer ;)
self.test_hook = test_hook
async def do_thing(
self, stream: "grpclib.server.Stream[DoThingRequest, DoThingResponse]"
):
request = await stream.recv_message()
if self.test_hook is not None:
self.test_hook(stream)
await stream.send_message(DoThingResponse([request.name]))
async def do_many_things(
self, stream: "grpclib.server.Stream[DoThingRequest, DoThingResponse]"
):
thing_names = [request.name for request in stream]
if self.test_hook is not None:
self.test_hook(stream)
await stream.send_message(DoThingResponse(thing_names))
async def get_thing_versions(
self, stream: "grpclib.server.Stream[GetThingRequest, GetThingResponse]"
):
request = await stream.recv_message()
if self.test_hook is not None:
self.test_hook(stream)
for version_num in range(1, 6):
await stream.send_message(
GetThingResponse(name=request.name, version=version_num)
)
async def get_different_things(
self, stream: "grpclib.server.Stream[GetThingRequest, GetThingResponse]"
):
if self.test_hook is not None:
self.test_hook(stream)
# Respond to each input item immediately
response_num = 0
async for request in stream:
response_num += 1
await stream.send_message(
GetThingResponse(name=request.name, version=response_num)
)
def __mapping__(self) -> Dict[str, "grpclib.const.Handler"]:
return {
"/service.Test/DoThing": grpclib.const.Handler(
self.do_thing,
grpclib.const.Cardinality.UNARY_UNARY,
DoThingRequest,
DoThingResponse,
),
"/service.Test/DoManyThings": grpclib.const.Handler(
self.do_many_things,
grpclib.const.Cardinality.STREAM_UNARY,
DoThingRequest,
DoThingResponse,
),
"/service.Test/GetThingVersions": grpclib.const.Handler(
self.get_thing_versions,
grpclib.const.Cardinality.UNARY_STREAM,
GetThingRequest,
GetThingResponse,
),
"/service.Test/GetDifferentThings": grpclib.const.Handler(
self.get_different_things,
grpclib.const.Cardinality.STREAM_STREAM,
GetThingRequest,
GetThingResponse,
),
}

View File

@@ -0,0 +1,6 @@
from betterproto.tests.output_betterproto.bool import Test
def test_value():
message = Test()
assert not message.value, "Boolean is False by default"

View File

@@ -9,4 +9,10 @@ enum my_enum {
message Test {
int32 camelCase = 1;
my_enum snake_case = 2;
snake_case_message snake_case_message = 3;
int32 UPPERCASE = 4;
}
message snake_case_message {
}

View File

@@ -0,0 +1,23 @@
import betterproto.tests.output_betterproto.casing as casing
from betterproto.tests.output_betterproto.casing import Test
def test_message_attributes():
message = Test()
assert hasattr(
message, "snake_case_message"
), "snake_case field name is same in python"
assert hasattr(message, "camel_case"), "CamelCase field is snake_case in python"
assert hasattr(message, "uppercase"), "UPPERCASE field is lowercase in python"
def test_message_casing():
assert hasattr(
casing, "SnakeCaseMessage"
), "snake_case Message name is converted to CamelCase in python"
def test_enum_casing():
assert hasattr(
casing, "MyEnum"
), "snake_case Enum name is converted to CamelCase in python"

View File

@@ -0,0 +1,7 @@
syntax = "proto3";
message Test {
int32 UPPERCASE = 1;
int32 UPPERCASE_V2 = 2;
int32 UPPER_CAMEL_CASE = 3;
}

View File

@@ -0,0 +1,14 @@
from betterproto.tests.output_betterproto.casing_message_field_uppercase import Test
def test_message_casing():
message = Test()
assert hasattr(
message, "uppercase"
), "UPPERCASE attribute is converted to 'uppercase' in python"
assert hasattr(
message, "uppercase_v2"
), "UPPERCASE_V2 attribute is converted to 'uppercase_v2' in python"
assert hasattr(
message, "upper_camel_case"
), "UPPER_CAMEL_CASE attribute is converted to upper_camel_case in python"

View File

@@ -0,0 +1,22 @@
# Test cases that are expected to fail, e.g. unimplemented features or bug-fixes.
# Remove from list when fixed.
xfail = {
"import_circular_dependency",
"oneof_enum", # 63
"namespace_keywords", # 70
"namespace_builtin_types", # 53
"googletypes_struct", # 9
"googletypes_value", # 9
"enum_skipped_value", # 93
"import_capitalized_package",
"example", # This is the example in the readme. Not a test.
}
services = {
"googletypes_response",
"googletypes_response_embedded",
"service",
"import_service_input_message",
"googletypes_service_returns_empty",
"googletypes_service_returns_googletype",
}

View File

@@ -0,0 +1,12 @@
syntax = "proto3";
message Test {
enum MyEnum {
ZERO = 0;
ONE = 1;
// TWO = 2;
THREE = 3;
FOUR = 4;
}
MyEnum x = 1;
}

View File

@@ -0,0 +1,18 @@
from betterproto.tests.output_betterproto.enum_skipped_value import (
Test,
TestMyEnum,
)
import pytest
@pytest.mark.xfail(reason="#93")
def test_message_attributes():
assert (
Test(x=TestMyEnum.ONE).to_dict()["x"] == "ONE"
), "MyEnum.ONE is not serialized to 'ONE'"
assert (
Test(x=TestMyEnum.THREE).to_dict()["x"] == "THREE"
), "MyEnum.THREE is not serialized to 'THREE'"
assert (
Test(x=TestMyEnum.FOUR).to_dict()["x"] == "FOUR"
), "MyEnum.FOUR is not serialized to 'FOUR'"

View File

@@ -0,0 +1,8 @@
syntax = "proto3";
package hello;
// Greeting represents a message you can tell a user.
message Greeting {
string message = 1;
}

View File

@@ -0,0 +1,6 @@
{
"foo": 4294967295,
"bar": -2147483648,
"baz": "18446744073709551615",
"qux": "-9223372036854775808"
}

View File

@@ -0,0 +1,8 @@
syntax = "proto3";
message Test {
fixed32 foo = 1;
sfixed32 bar = 2;
fixed64 baz = 3;
sfixed64 qux = 4;
}

View File

@@ -1,5 +1,7 @@
{
"maybe": false,
"ts": "1972-01-01T10:00:20.021Z",
"duration": "1.200s"
"duration": "1.200s",
"important": 10,
"empty": {}
}

View File

@@ -3,10 +3,12 @@ syntax = "proto3";
import "google/protobuf/duration.proto";
import "google/protobuf/timestamp.proto";
import "google/protobuf/wrappers.proto";
import "google/protobuf/empty.proto";
message Test {
google.protobuf.BoolValue maybe = 1;
google.protobuf.Timestamp ts = 2;
google.protobuf.Duration duration = 3;
google.protobuf.Int32Value important = 4;
google.protobuf.Empty empty = 5;
}

View File

@@ -0,0 +1,21 @@
syntax = "proto3";
import "google/protobuf/wrappers.proto";
// Tests that wrapped values can be used directly as return values
service Test {
rpc GetDouble (Input) returns (google.protobuf.DoubleValue);
rpc GetFloat (Input) returns (google.protobuf.FloatValue);
rpc GetInt64 (Input) returns (google.protobuf.Int64Value);
rpc GetUInt64 (Input) returns (google.protobuf.UInt64Value);
rpc GetInt32 (Input) returns (google.protobuf.Int32Value);
rpc GetUInt32 (Input) returns (google.protobuf.UInt32Value);
rpc GetBool (Input) returns (google.protobuf.BoolValue);
rpc GetString (Input) returns (google.protobuf.StringValue);
rpc GetBytes (Input) returns (google.protobuf.BytesValue);
}
message Input {
}

View File

@@ -0,0 +1,54 @@
from typing import Any, Callable, Optional
import betterproto.lib.google.protobuf as protobuf
import pytest
from betterproto.tests.mocks import MockChannel
from betterproto.tests.output_betterproto.googletypes_response import TestStub
test_cases = [
(TestStub.get_double, protobuf.DoubleValue, 2.5),
(TestStub.get_float, protobuf.FloatValue, 2.5),
(TestStub.get_int64, protobuf.Int64Value, -64),
(TestStub.get_u_int64, protobuf.UInt64Value, 64),
(TestStub.get_int32, protobuf.Int32Value, -32),
(TestStub.get_u_int32, protobuf.UInt32Value, 32),
(TestStub.get_bool, protobuf.BoolValue, True),
(TestStub.get_string, protobuf.StringValue, "string"),
(TestStub.get_bytes, protobuf.BytesValue, bytes(0xFF)[0:4]),
]
@pytest.mark.asyncio
@pytest.mark.parametrize(["service_method", "wrapper_class", "value"], test_cases)
async def test_channel_recieves_wrapped_type(
service_method: Callable[[TestStub], Any], wrapper_class: Callable, value
):
wrapped_value = wrapper_class()
wrapped_value.value = value
channel = MockChannel(responses=[wrapped_value])
service = TestStub(channel)
await service_method(service)
assert channel.requests[0]["response_type"] != Optional[type(value)]
assert channel.requests[0]["response_type"] == type(wrapped_value)
@pytest.mark.asyncio
@pytest.mark.xfail
@pytest.mark.parametrize(["service_method", "wrapper_class", "value"], test_cases)
async def test_service_unwraps_response(
service_method: Callable[[TestStub], Any], wrapper_class: Callable, value
):
"""
grpclib does not unwrap wrapper values returned by services
"""
wrapped_value = wrapper_class()
wrapped_value.value = value
service = TestStub(MockChannel(responses=[wrapped_value]))
response_value = await service_method(service)
assert response_value == value
assert type(response_value) == type(value)

View File

@@ -0,0 +1,24 @@
syntax = "proto3";
import "google/protobuf/wrappers.proto";
// Tests that wrapped values are supported as part of output message
service Test {
rpc getOutput (Input) returns (Output);
}
message Input {
}
message Output {
google.protobuf.DoubleValue double_value = 1;
google.protobuf.FloatValue float_value = 2;
google.protobuf.Int64Value int64_value = 3;
google.protobuf.UInt64Value uint64_value = 4;
google.protobuf.Int32Value int32_value = 5;
google.protobuf.UInt32Value uint32_value = 6;
google.protobuf.BoolValue bool_value = 7;
google.protobuf.StringValue string_value = 8;
google.protobuf.BytesValue bytes_value = 9;
}

View File

@@ -0,0 +1,39 @@
import pytest
from betterproto.tests.mocks import MockChannel
from betterproto.tests.output_betterproto.googletypes_response_embedded import (
Output,
TestStub,
)
@pytest.mark.asyncio
async def test_service_passes_through_unwrapped_values_embedded_in_response():
"""
We do not not need to implement value unwrapping for embedded well-known types,
as this is already handled by grpclib. This test merely shows that this is the case.
"""
output = Output(
double_value=10.0,
float_value=12.0,
int64_value=-13,
uint64_value=14,
int32_value=-15,
uint32_value=16,
bool_value=True,
string_value="string",
bytes_value=bytes(0xFF)[0:4],
)
service = TestStub(MockChannel(responses=[output]))
response = await service.get_output()
assert response.double_value == 10.0
assert response.float_value == 12.0
assert response.int64_value == -13
assert response.uint64_value == 14
assert response.int32_value == -15
assert response.uint32_value == 16
assert response.bool_value
assert response.string_value == "string"
assert response.bytes_value == bytes(0xFF)[0:4]

View File

@@ -0,0 +1,11 @@
syntax = "proto3";
import "google/protobuf/empty.proto";
service Test {
rpc Send (RequestMessage) returns (google.protobuf.Empty) {
}
}
message RequestMessage {
}

View File

@@ -0,0 +1,16 @@
syntax = "proto3";
import "google/protobuf/empty.proto";
import "google/protobuf/struct.proto";
// Tests that imports are generated correctly when returning Google well-known types
service Test {
rpc GetEmpty (RequestMessage) returns (google.protobuf.Empty);
rpc GetStruct (RequestMessage) returns (google.protobuf.Struct);
rpc GetListValue (RequestMessage) returns (google.protobuf.ListValue);
rpc GetValue (RequestMessage) returns (google.protobuf.Value);
}
message RequestMessage {
}

View File

@@ -0,0 +1,5 @@
{
"struct": {
"key": true
}
}

View File

@@ -0,0 +1,7 @@
syntax = "proto3";
import "google/protobuf/struct.proto";
message Test {
google.protobuf.Struct struct = 1;
}

View File

@@ -0,0 +1,11 @@
{
"value1": "hello world",
"value2": true,
"value3": 1,
"value4": null,
"value5": [
1,
2,
3
]
}

View File

@@ -0,0 +1,13 @@
syntax = "proto3";
import "google/protobuf/struct.proto";
// Tests that fields of type google.protobuf.Value can contain arbitrary JSON-values.
message Test {
google.protobuf.Value value1 = 1;
google.protobuf.Value value2 = 2;
google.protobuf.Value value3 = 3;
google.protobuf.Value value4 = 4;
google.protobuf.Value value5 = 5;
}

View File

@@ -0,0 +1,8 @@
syntax = "proto3";
package Capitalized;
message Message {
}

View File

@@ -0,0 +1,9 @@
syntax = "proto3";
import "capitalized.proto";
// Tests that we can import from a package with a capital name, that looks like a nested type, but isn't.
message Test {
Capitalized.Message message = 1;
}

View File

@@ -0,0 +1,7 @@
syntax = "proto3";
package package.childpackage;
message ChildMessage {
}

View File

@@ -0,0 +1,9 @@
syntax = "proto3";
import "package_message.proto";
// Tests generated imports when a message in a package refers to a message in a nested child package.
message Test {
package.PackageMessage message = 1;
}

View File

@@ -0,0 +1,9 @@
syntax = "proto3";
import "child.proto";
package package;
message PackageMessage {
package.childpackage.ChildMessage c = 1;
}

View File

@@ -0,0 +1,7 @@
syntax = "proto3";
package childpackage;
message Message {
}

View File

@@ -0,0 +1,9 @@
syntax = "proto3";
import "child.proto";
// Tests generated imports when a message in root refers to a message in a child package.
message Test {
childpackage.Message child = 1;
}

View File

@@ -0,0 +1,28 @@
syntax = "proto3";
import "root.proto";
import "other.proto";
// This test-case verifies support for circular dependencies in the generated python files.
//
// This is important because we generate 1 python file/module per package, rather than 1 file per proto file.
//
// Scenario:
//
// The proto messages depend on each other in a non-circular way:
//
// Test -------> RootPackageMessage <--------------.
// `------------------------------------> OtherPackageMessage
//
// Test and RootPackageMessage are in different files, but belong to the same package (root):
//
// (Test -------> RootPackageMessage) <------------.
// `------------------------------------> OtherPackageMessage
//
// After grouping the packages into single files or modules, a circular dependency is created:
//
// (root: Test & RootPackageMessage) <-------> (other: OtherPackageMessage)
message Test {
RootPackageMessage message = 1;
other.OtherPackageMessage other = 2;
}

View File

@@ -0,0 +1,8 @@
syntax = "proto3";
import "root.proto";
package other;
message OtherPackageMessage {
RootPackageMessage rootPackageMessage = 1;
}

View File

@@ -0,0 +1,5 @@
syntax = "proto3";
message RootPackageMessage {
}

View File

@@ -0,0 +1,6 @@
syntax = "proto3";
package cousin.cousin_subpackage;
message CousinMessage {
}

View File

@@ -0,0 +1,11 @@
syntax = "proto3";
package test.subpackage;
import "cousin.proto";
// Verify that we can import message unrelated to us
message Test {
cousin.cousin_subpackage.CousinMessage message = 1;
}

View File

@@ -0,0 +1,6 @@
syntax = "proto3";
package cousin.subpackage;
message CousinMessage {
}

View File

@@ -0,0 +1,11 @@
syntax = "proto3";
package test.subpackage;
import "cousin.proto";
// Verify that we can import a message unrelated to us, in a subpackage with the same name as us.
message Test {
cousin.subpackage.CousinMessage message = 1;
}

View File

@@ -0,0 +1,11 @@
syntax = "proto3";
import "users_v1.proto";
import "posts_v1.proto";
// Tests generated message can correctly reference two packages with the same leaf-name
message Test {
users.v1.User user = 1;
posts.v1.Post post = 2;
}

View File

@@ -0,0 +1,7 @@
syntax = "proto3";
package posts.v1;
message Post {
}

View File

@@ -0,0 +1,7 @@
syntax = "proto3";
package users.v1;
message User {
}

View File

@@ -0,0 +1,12 @@
syntax = "proto3";
import "parent_package_message.proto";
package parent.child;
// Tests generated imports when a message refers to a message defined in its parent package
message Test {
ParentPackageMessage message_implicit = 1;
parent.ParentPackageMessage message_explicit = 2;
}

View File

@@ -0,0 +1,6 @@
syntax = "proto3";
package parent;
message ParentPackageMessage {
}

View File

@@ -0,0 +1,11 @@
syntax = "proto3";
package child;
import "root.proto";
// Verify that we can import root message from child package
message Test {
RootMessage message = 1;
}

View File

@@ -0,0 +1,5 @@
syntax = "proto3";
message RootMessage {
}

View File

@@ -0,0 +1,9 @@
syntax = "proto3";
import "sibling.proto";
// Tests generated imports when a message in the root package refers to another message in the root package
message Test {
SiblingMessage sibling = 1;
}

View File

@@ -0,0 +1,5 @@
syntax = "proto3";
message SiblingMessage {
}

View File

@@ -0,0 +1,15 @@
syntax = "proto3";
import "request_message.proto";
// Tests generated service correctly imports the RequestMessage
service Test {
rpc DoThing (RequestMessage) returns (RequestResponse);
}
message RequestResponse {
int32 value = 1;
}

View File

@@ -0,0 +1,5 @@
syntax = "proto3";
message RequestMessage {
int32 argument = 1;
}

View File

@@ -0,0 +1,16 @@
import pytest
from betterproto.tests.mocks import MockChannel
from betterproto.tests.output_betterproto.import_service_input_message import (
RequestResponse,
TestStub,
)
@pytest.mark.xfail(reason="#68 Request Input Messages are not imported for service")
@pytest.mark.asyncio
async def test_service_correctly_imports_reference_message():
mock_response = RequestResponse(value=10)
service = TestStub(MockChannel([mock_response]))
response = await service.do_thing()
assert mock_response == response

View File

@@ -0,0 +1,4 @@
{
"positive": 150,
"negative": -150
}

View File

@@ -3,5 +3,6 @@ syntax = "proto3";
// Some documentation about the Test message.
message Test {
// Some documentation about the count.
int32 count = 1;
int32 positive = 1;
int32 negative = 2;
}

View File

@@ -0,0 +1,16 @@
{
"int": "value-for-int",
"float": "value-for-float",
"complex": "value-for-complex",
"list": "value-for-list",
"tuple": "value-for-tuple",
"range": "value-for-range",
"str": "value-for-str",
"bytearray": "value-for-bytearray",
"bytes": "value-for-bytes",
"memoryview": "value-for-memoryview",
"set": "value-for-set",
"frozenset": "value-for-frozenset",
"map": "value-for-map",
"bool": "value-for-bool"
}

View File

@@ -0,0 +1,38 @@
syntax = "proto3";
// Tests that messages may contain fields with names that are python types
message Test {
// https://docs.python.org/2/library/stdtypes.html#numeric-types-int-float-long-complex
string int = 1;
string float = 2;
string complex = 3;
// https://docs.python.org/3/library/stdtypes.html#sequence-types-list-tuple-range
string list = 4;
string tuple = 5;
string range = 6;
// https://docs.python.org/3/library/stdtypes.html#str
string str = 7;
// https://docs.python.org/3/library/stdtypes.html#bytearray-objects
string bytearray = 8;
// https://docs.python.org/3/library/stdtypes.html#bytes-and-bytearray-operations
string bytes = 9;
// https://docs.python.org/3/library/stdtypes.html#memory-views
string memoryview = 10;
// https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset
string set = 11;
string frozenset = 12;
// https://docs.python.org/3/library/stdtypes.html#dict
string map = 13;
string dict = 14;
// https://docs.python.org/3/library/stdtypes.html#boolean-values
string bool = 15;
}

View File

@@ -0,0 +1,37 @@
{
"False": 1,
"None": 2,
"True": 3,
"and": 4,
"as": 5,
"assert": 6,
"async": 7,
"await": 8,
"break": 9,
"class": 10,
"continue": 11,
"def": 12,
"del": 13,
"elif": 14,
"else": 15,
"except": 16,
"finally": 17,
"for": 18,
"from": 19,
"global": 20,
"if": 21,
"import": 22,
"in": 23,
"is": 24,
"lambda": 25,
"nonlocal": 26,
"not": 27,
"or": 28,
"pass": 29,
"raise": 30,
"return": 31,
"try": 32,
"while": 33,
"with": 34,
"yield": 35
}

Some files were not shown because too many files have changed in this diff Show More