This project aims to provide an improved experience when using Protobuf / gRPC in a modern Python environment by making use of modern language features and generating readable, understandable, idiomatic Python code. It will not support legacy features or environments. The following are supported:

Protobuf 3 & gRPC code generation
- Both binary & JSON serialization is built-in
Python 3.7+ making use of:
- Enums
- Dataclasses
- async/await
- Relative imports
- Mypy type checking

This project is heavily inspired by, and borrows functionality from:

Motivation

This project exists because I am unhappy with the state of the official Google protoc plugin for Python.

No async support (requires additional grpclib plugin)
No typing support or code completion/intelligence (requires additional mypy plugin)
No __init__.py module files get generated
Output is not importable
- Import paths break in Python 3 unless you mess with sys.path
Bugs when names clash (e.g. codecs package)
Generated code is not idiomatic
- Completely unreadable runtime code-generation
- Much code looks like C++ or Java ported 1:1 to Python
- Capitalized function names like HasField() and SerializeToString()
- Uses SerializeToString() rather than the built-in __bytes__()

This project is a reimplementation from the ground up focused on idiomatic modern Python to help fix some of the above. While it may not be a 1:1 drop-in replacement due to changed method names and call patterns, the wire format is identical.

Installation & Getting Started

First, install the package:

$ pip install betterproto

Now, given a proto file, e.g example.proto:

syntax = "proto3";

package hello;

// Greeting represents a message you can tell a user.
message Greeting {
  string message = 1;
}

You can run the following:

$ protoc -I . --python_betterproto_out=. example.proto

This will generate hello.py which looks like:

# Generated by the protocol buffer compiler.  DO NOT EDIT!
# sources: hello.proto
# plugin: python-betterproto
from dataclasses import dataclass

import betterproto


@dataclass
class Hello(betterproto.Message):
    """Greeting represents a message you can tell a user."""

    message: str = betterproto.string_field(1)

Now you can use it!

>>> from hello import Hello
>>> test = Hello()
>>> test
Hello(message='')

>>> test.message = "Hey!"
>>> test
Hello(message="Hey!")

>>> serialized = bytes(test)
>>> serialized
b'\n\x04Hey!'

>>> another = Hello().parse(serialized)
>>> another
Hello(message="Hey!")

>>> another.to_dict()
{"message": "Hey!"}
>>> another.to_json(indent=2)
'{\n  "message": "Hey!"\n}'

Async gRPC Support

The generated Protobuf Message classes are compatible with grpclib so you are free to use it if you like. That said, this project also includes support for async gRPC stub generation with better static type checking and code completion support. It is enabled by default.

Given an example like:

syntax = "proto3";

package echo;

message EchoRequest {
  string value = 1;
  // Number of extra times to echo
  uint32 extra_times = 2;
}

message EchoResponse {
  repeated string values = 1;
}

message EchoStreamResponse  {
  string value = 1;
}

service Echo {
  rpc Echo(EchoRequest) returns (EchoResponse);
  rpc EchoStream(EchoRequest) returns (stream EchoStreamResponse);
}

You can use it like so (enable async in the interactive shell first):

>>> import echo
>>> from grpclib.client import Channel

>>> channel = Channel(host="127.0.0.1", port=1234)
>>> service = echo.EchoStub(channel)
>>> await service.echo(value="hello", extra_times=1)
EchoResponse(values=["hello", "hello"])

>>> async for response in service.echo_stream(value="hello", extra_times=1)
    print(response)

EchoStreamResponse(value="hello")
EchoStreamResponse(value="hello")

JSON

Both serializing and parsing are supported to/from JSON and Python dictionaries using the following methods:

Dicts: Message().to_dict(), Message().from_dict(...)
JSON: Message().to_json(), Message().from_json(...)

Determining if a message was sent

Sometimes it is useful to be able to determine whether a message has been sent on the wire. This is how the Google wrapper types work to let you know whether a value is unset, set as the default (zero value), or set as something else, for example.

Use Message().serialized_on_wire to determine if it was sent. This is a little bit different from the official Google generated Python code:

# Old way (official Google Protobuf package)
>>> mymessage.HasField('myfield')

# New way (this project)
>>> mymessage.myfield.serialized_on_wire

Development

First, make sure you have Python 3.7+ and pipenv installed:

# Get set up with the virtual env & dependencies
$ pipenv install --dev

# Link the local package
$ pipenv shell
$ pip install -e .

Tests

There are two types of tests:

Manually-written tests for some behavior of the library
Proto files and JSON inputs for automated tests

For #2, you can add a new *.proto file into the betterproto/tests directory along with a sample *.json input and it will get automatically picked up.

Here's how to run the tests.

# Generate assets from sample .proto files
$ pipenv run generate

# Run the tests
$ pipenv run tests

TODO

Fixed length fields
- Packed fixed-length
Zig-zag signed fields (sint32, sint64)
Don't encode zero values for nested types
Enums
Repeated message fields
Maps
- Maps of message fields
Support passthrough of unknown fields
Refs to nested types
Imports in proto files
Well-known Google types
JSON that isn't completely naive.
- 64-bit ints as strings
- Maps
- Lists
- Bytes as base64
- Any support
- Enum strings
- Well known types support (timestamp, duration, wrappers)
Async service stubs
- Unary-unary
- Server streaming response
- Client streaming request
Renaming messages and fields to conform to Python name standards
Renaming clashes with language keywords and standard library top-level packages
Python package
Automate running tests
Cleanup!

License

http://dgt.mit-license.org/