Fixes circular import problem when a non-circular dependency triangle is flattened into two python packages

CHANGELOG.md

@@ -5,6 +5,28 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+- Versions suffixed with `b*` are in `beta` and can be installed with `pip install --pre betterproto`.
+
+## [2.0.0b1] - 2020-07-04
+
+[Upgrade Guide](./docs/upgrading.md) 
+
+> Several bugfixes and improvements required or will require small breaking changes, necessitating a new version.
+> `2.0.0` will be released once the interface is stable.
+
+- Add support for gRPC  and **stream-stream** [#83](https://github.com/danielgtaylor/python-betterproto/pull/83)
+- Switch from  to `poetry` for development [#75](https://github.com/danielgtaylor/python-betterproto/pull/75)
+- Fix No arguments are generated for stub methods when using import with proto definition 
+- Fix two packages with the same name suffix should not cause naming conflict [#25](https://github.com/danielgtaylor/python-betterproto/issues/25)
+
+- Fix Import child package from root [#57](https://github.com/danielgtaylor/python-betterproto/issues/57)
+- Fix Import child package from package [#58](https://github.com/danielgtaylor/python-betterproto/issues/58)
+- Fix Import parent package from child package [#59](https://github.com/danielgtaylor/python-betterproto/issues/59)
+- Fix Import root package from child package [#60](https://github.com/danielgtaylor/python-betterproto/issues/60)
+- Fix Import root package from root [#61](https://github.com/danielgtaylor/python-betterproto/issues/61)
+
+- Fix ALL_CAPS message fields are parsed incorrectly. [#11](https://github.com/danielgtaylor/python-betterproto/issues/11)
+
 ## [1.2.5] - 2020-04-27
 
 - Add .j2 suffix to python template names to avoid confusing certain build tools [#72](https://github.com/danielgtaylor/python-betterproto/pull/72)

README.md

@@ -40,7 +40,7 @@ This project exists because I am unhappy with the state of the official Google p
 
 This project is a reimplementation from the ground up focused on idiomatic modern Python to help fix some of the above. While it may not be a 1:1 drop-in replacement due to changed method names and call patterns, the wire format is identical.
 
-## Installation & Getting Started
+## Installation
 
 First, install the package. Note that the `[compiler]` feature flag tells it to install extra dependencies only needed by the `protoc` plugin:
 
@@ -52,6 +52,12 @@ pip install "betterproto[compiler]"
 pip install betterproto
 ```
 
+*Betterproto* is under active development. To install the latest beta version, use `pip install --pre betterproto`.
+
+## Getting Started
+
+### Compiling proto files
+
 Now, given you installed the compiler and have a proto file, e.g `example.proto`:
 
 ```protobuf
@@ -149,7 +155,7 @@ service Echo {
 
 You can use it like so (enable async in the interactive shell first):
 
-```py
+```python
 >>> import echo
 >>> from grpclib.client import Channel
 
@@ -174,8 +180,8 @@ Both serializing and parsing are supported to/from JSON and Python dictionaries
 
 For compatibility the default is to convert field names to `camelCase`. You can control this behavior by passing a casing value, e.g:
 
-```py
+```python
->>> MyMessage().to_dict(casing=betterproto.Casing.SNAKE)
+MyMessage().to_dict(casing=betterproto.Casing.SNAKE)
 ```
 
 ### Determining if a message was sent
@@ -300,7 +306,22 @@ datetime.datetime(2019, 1, 1, 11, 59, 58, 800000, tzinfo=datetime.timezone.utc)
 
 Join us on [Slack](https://join.slack.com/t/betterproto/shared_invite/zt-f0n0uolx-iN8gBNrkPxtKHTLpG3o1OQ)!
 
-First, make sure you have Python 3.6+ and `poetry` installed, along with the official [Protobuf Compiler](https://github.com/protocolbuffers/protobuf/releases) for your platform. Then:
+### Requirements
+
+- Python (3.6 or higher)
+
+- [protoc](https://grpc.io/docs/protoc-installation/) (3.12 or higher)
+  *Needed to compile `.proto` files and run the tests*
+
+- [poetry](https://python-poetry.org/docs/#installation)
+  *Needed to install dependencies in a virtual environment*
+
+- make ([ubuntu](https://www.howtoinstall.me/ubuntu/18-04/make/), [windows](https://stackoverflow.com/questions/32127524/how-to-install-and-use-make-in-windows), [mac](https://osxdaily.com/2014/02/12/install-command-line-tools-mac-os-x/))
+
+  *Needed to conveniently run development tasks.*
+  *Alternatively, manually run the commands defined in the [Makefile](./Makefile)*
+
+### Setup
 
 ```sh
 # Get set up with the virtual env & dependencies
@@ -310,7 +331,7 @@ poetry install
 poetry shell
 ```
 
-To benefit from the collection of standard development tasks ensure you have make installed and run `make help` to see available tasks.
+Run `make help` to see all available development tasks.
 
 ### Code style
 

betterproto/compile/importing.py

@@ -86,7 +86,7 @@ def reference_absolute(imports, py_package, py_type):
     string_import = ".".join(py_package)
     string_alias = safe_snake_case(string_import)
     imports.add(f"import {string_import} as {string_alias}")
-    return f"{string_alias}.{py_type}"
+    return f'"{string_alias}.{py_type}"'
 
 
 def reference_sibling(py_type: str) -> str:
@@ -109,10 +109,10 @@ def reference_descendent(
     if string_from:
         string_alias = "_".join(importing_descendent)
         imports.add(f"from .{string_from} import {string_import} as {string_alias}")
-        return f"{string_alias}.{py_type}"
+        return f'"{string_alias}.{py_type}"'
     else:
         imports.add(f"from . import {string_import}")
-        return f"{string_import}.{py_type}"
+        return f'"{string_import}.{py_type}"'
 
 
 def reference_ancestor(
@@ -130,11 +130,11 @@ def reference_ancestor(
         string_alias = f"_{'_' * distance_up}{string_import}__"
         string_from = f"..{'.' * distance_up}"
         imports.add(f"from {string_from} import {string_import} as {string_alias}")
-        return f"{string_alias}.{py_type}"
+        return f'"{string_alias}.{py_type}"'
     else:
         string_alias = f"{'_' * distance_up}{py_type}__"
         imports.add(f"from .{'.' * distance_up} import {py_type} as {string_alias}")
-        return string_alias
+        return f'"{string_alias}"'
 
 
 def reference_cousin(
@@ -157,4 +157,4 @@ def reference_cousin(
         + "__"
     )
     imports.add(f"from {string_from} import {string_import} as {string_alias}")
-    return f"{string_alias}.{py_type}"
+    return f'"{string_alias}.{py_type}"'

betterproto/plugin.py

@@ -329,7 +329,7 @@ def generate_code(request, response):
                                 output["imports"],
                                 method.output_type,
                                 unwrap=False,
-                            ).strip('"'),
+                            ),
                             "client_streaming": method.client_streaming,
                             "server_streaming": method.server_streaming,
                         }

betterproto/templates/template.py.j2

@@ -16,10 +16,6 @@ import betterproto
 import grpclib
 {% endif %}
 
-{% for i in description.imports %}
-{{ i }}
-{% endfor %}
-
 
 {% if description.enums %}{% for enum in description.enums %}
 class {{ enum.py_name }}(betterproto.Enum):
@@ -102,14 +98,14 @@ class {{ service.py_name }}Stub(betterproto.ServiceStub):
             "{{ method.route }}",
             request_iterator,
             {{ method.input }},
-            {{ method.output }},
+            {{ method.output.strip('"') }},
         ):
             yield response
             {% else %}{# i.e. not client streaming #}
         async for response in self._unary_stream(
             "{{ method.route }}",
             request,
-            {{ method.output }},
+            {{ method.output.strip('"') }},
         ):
             yield response
 
@@ -120,16 +116,20 @@ class {{ service.py_name }}Stub(betterproto.ServiceStub):
             "{{ method.route }}",
             request_iterator,
             {{ method.input }},
-            {{ method.output }}
+            {{ method.output.strip('"') }}
         )
             {% else %}{# i.e. not client streaming #}
         return await self._unary_unary(
             "{{ method.route }}",
             request,
-            {{ method.output }}
+            {{ method.output.strip('"') }}
         )
             {% endif %}{# client streaming #}
         {% endif %}
 
     {% endfor %}
 {% endfor %}
+
+{% for i in description.imports %}
+{{ i }}
+{% endfor %}

betterproto/tests/inputs/config.py

@@ -1,13 +1,11 @@
 # Test cases that are expected to fail, e.g. unimplemented features or bug-fixes.
 # Remove from list when fixed.
 xfail = {
-    "import_circular_dependency",
     "oneof_enum",  # 63
     "namespace_keywords",  # 70
     "namespace_builtin_types",  # 53
     "googletypes_struct",  # 9
-    "googletypes_value",  # 9
+    "googletypes_value",  # 9,
-    "enum_skipped_value",  # 93
     "import_capitalized_package",
     "example",  # This is the example in the readme. Not a test.
 }

betterproto/tests/inputs/enum_skipped_value/enum_skipped_value.proto

@@ -1,12 +0,0 @@
-syntax = "proto3";
-
-message Test {
-  enum MyEnum {
-    ZERO = 0;
-    ONE = 1;
-    // TWO = 2;
-    THREE = 3;
-    FOUR = 4;
-  }
-  MyEnum x = 1;
-}

betterproto/tests/inputs/enum_skipped_value/test_enum_skipped_value.py

@@ -1,18 +0,0 @@
-from betterproto.tests.output_betterproto.enum_skipped_value import (
-    Test,
-    TestMyEnum,
-)
-import pytest
-
-
-@pytest.mark.xfail(reason="#93")
-def test_message_attributes():
-    assert (
-        Test(x=TestMyEnum.ONE).to_dict()["x"] == "ONE"
-    ), "MyEnum.ONE is not serialized to 'ONE'"
-    assert (
-        Test(x=TestMyEnum.THREE).to_dict()["x"] == "THREE"
-    ), "MyEnum.THREE is not serialized to 'THREE'"
-    assert (
-        Test(x=TestMyEnum.FOUR).to_dict()["x"] == "FOUR"
-    ), "MyEnum.FOUR is not serialized to 'FOUR'"

betterproto/tests/test_get_ref_type.py

@@ -8,22 +8,22 @@ from ..compile.importing import get_type_reference, parse_source_type_name
     [
         (
             ".google.protobuf.Empty",
-            "betterproto_lib_google_protobuf.Empty",
+            '"betterproto_lib_google_protobuf.Empty"',
             "import betterproto.lib.google.protobuf as betterproto_lib_google_protobuf",
         ),
         (
             ".google.protobuf.Struct",
-            "betterproto_lib_google_protobuf.Struct",
+            '"betterproto_lib_google_protobuf.Struct"',
             "import betterproto.lib.google.protobuf as betterproto_lib_google_protobuf",
         ),
         (
             ".google.protobuf.ListValue",
-            "betterproto_lib_google_protobuf.ListValue",
+            '"betterproto_lib_google_protobuf.ListValue"',
             "import betterproto.lib.google.protobuf as betterproto_lib_google_protobuf",
         ),
         (
             ".google.protobuf.Value",
-            "betterproto_lib_google_protobuf.Value",
+            '"betterproto_lib_google_protobuf.Value"',
             "import betterproto.lib.google.protobuf as betterproto_lib_google_protobuf",
         ),
     ],
@@ -67,15 +67,27 @@ def test_referenceing_google_wrappers_unwraps_them(
 @pytest.mark.parametrize(
     ["google_type", "expected_name"],
     [
-        (".google.protobuf.DoubleValue", "betterproto_lib_google_protobuf.DoubleValue"),
+        (
-        (".google.protobuf.FloatValue", "betterproto_lib_google_protobuf.FloatValue"),
+            ".google.protobuf.DoubleValue",
-        (".google.protobuf.Int32Value", "betterproto_lib_google_protobuf.Int32Value"),
+            '"betterproto_lib_google_protobuf.DoubleValue"',
-        (".google.protobuf.Int64Value", "betterproto_lib_google_protobuf.Int64Value"),
+        ),
-        (".google.protobuf.UInt32Value", "betterproto_lib_google_protobuf.UInt32Value"),
+        (".google.protobuf.FloatValue", '"betterproto_lib_google_protobuf.FloatValue"'),
-        (".google.protobuf.UInt64Value", "betterproto_lib_google_protobuf.UInt64Value"),
+        (".google.protobuf.Int32Value", '"betterproto_lib_google_protobuf.Int32Value"'),
-        (".google.protobuf.BoolValue", "betterproto_lib_google_protobuf.BoolValue"),
+        (".google.protobuf.Int64Value", '"betterproto_lib_google_protobuf.Int64Value"'),
-        (".google.protobuf.StringValue", "betterproto_lib_google_protobuf.StringValue"),
+        (
-        (".google.protobuf.BytesValue", "betterproto_lib_google_protobuf.BytesValue"),
+            ".google.protobuf.UInt32Value",
+            '"betterproto_lib_google_protobuf.UInt32Value"',
+        ),
+        (
+            ".google.protobuf.UInt64Value",
+            '"betterproto_lib_google_protobuf.UInt64Value"',
+        ),
+        (".google.protobuf.BoolValue", '"betterproto_lib_google_protobuf.BoolValue"'),
+        (
+            ".google.protobuf.StringValue",
+            '"betterproto_lib_google_protobuf.StringValue"',
+        ),
+        (".google.protobuf.BytesValue", '"betterproto_lib_google_protobuf.BytesValue"'),
     ],
 )
 def test_referenceing_google_wrappers_without_unwrapping(
@@ -95,7 +107,7 @@ def test_reference_child_package_from_package():
     )
 
     assert imports == {"from . import child"}
-    assert name == "child.Message"
+    assert name == '"child.Message"'
 
 
 def test_reference_child_package_from_root():
@@ -103,7 +115,7 @@ def test_reference_child_package_from_root():
     name = get_type_reference(package="", imports=imports, source_type="child.Message")
 
     assert imports == {"from . import child"}
-    assert name == "child.Message"
+    assert name == '"child.Message"'
 
 
 def test_reference_camel_cased():
@@ -113,7 +125,7 @@ def test_reference_camel_cased():
     )
 
     assert imports == {"from . import child_package"}
-    assert name == "child_package.ExampleMessage"
+    assert name == '"child_package.ExampleMessage"'
 
 
 def test_reference_nested_child_from_root():
@@ -123,7 +135,7 @@ def test_reference_nested_child_from_root():
     )
 
     assert imports == {"from .nested import child as nested_child"}
-    assert name == "nested_child.Message"
+    assert name == '"nested_child.Message"'
 
 
 def test_reference_deeply_nested_child_from_root():
@@ -133,7 +145,7 @@ def test_reference_deeply_nested_child_from_root():
     )
 
     assert imports == {"from .deeply.nested import child as deeply_nested_child"}
-    assert name == "deeply_nested_child.Message"
+    assert name == '"deeply_nested_child.Message"'
 
 
 def test_reference_deeply_nested_child_from_package():
@@ -145,7 +157,7 @@ def test_reference_deeply_nested_child_from_package():
     )
 
     assert imports == {"from .deeply.nested import child as deeply_nested_child"}
-    assert name == "deeply_nested_child.Message"
+    assert name == '"deeply_nested_child.Message"'
 
 
 def test_reference_root_sibling():
@@ -181,7 +193,7 @@ def test_reference_parent_package_from_child():
     )
 
     assert imports == {"from ... import package as __package__"}
-    assert name == "__package__.Message"
+    assert name == '"__package__.Message"'
 
 
 def test_reference_parent_package_from_deeply_nested_child():
@@ -193,7 +205,7 @@ def test_reference_parent_package_from_deeply_nested_child():
     )
 
     assert imports == {"from ... import nested as __nested__"}
-    assert name == "__nested__.Message"
+    assert name == '"__nested__.Message"'
 
 
 def test_reference_ancestor_package_from_nested_child():
@@ -205,7 +217,7 @@ def test_reference_ancestor_package_from_nested_child():
     )
 
     assert imports == {"from .... import ancestor as ___ancestor__"}
-    assert name == "___ancestor__.Message"
+    assert name == '"___ancestor__.Message"'
 
 
 def test_reference_root_package_from_child():
@@ -215,7 +227,7 @@ def test_reference_root_package_from_child():
     )
 
     assert imports == {"from ... import Message as __Message__"}
-    assert name == "__Message__"
+    assert name == '"__Message__"'
 
 
 def test_reference_root_package_from_deeply_nested_child():
@@ -225,7 +237,7 @@ def test_reference_root_package_from_deeply_nested_child():
     )
 
     assert imports == {"from ..... import Message as ____Message__"}
-    assert name == "____Message__"
+    assert name == '"____Message__"'
 
 
 def test_reference_unrelated_package():
@@ -233,7 +245,7 @@ def test_reference_unrelated_package():
     name = get_type_reference(package="a", imports=imports, source_type="p.Message")
 
     assert imports == {"from .. import p as _p__"}
-    assert name == "_p__.Message"
+    assert name == '"_p__.Message"'
 
 
 def test_reference_unrelated_nested_package():
@@ -241,7 +253,7 @@ def test_reference_unrelated_nested_package():
     name = get_type_reference(package="a.b", imports=imports, source_type="p.q.Message")
 
     assert imports == {"from ...p import q as __p_q__"}
-    assert name == "__p_q__.Message"
+    assert name == '"__p_q__.Message"'
 
 
 def test_reference_unrelated_deeply_nested_package():
@@ -251,7 +263,7 @@ def test_reference_unrelated_deeply_nested_package():
     )
 
     assert imports == {"from .....p.q.r import s as ____p_q_r_s__"}
-    assert name == "____p_q_r_s__.Message"
+    assert name == '"____p_q_r_s__.Message"'
 
 
 def test_reference_cousin_package():
@@ -259,7 +271,7 @@ def test_reference_cousin_package():
     name = get_type_reference(package="a.x", imports=imports, source_type="a.y.Message")
 
     assert imports == {"from .. import y as _y__"}
-    assert name == "_y__.Message"
+    assert name == '"_y__.Message"'
 
 
 def test_reference_cousin_package_different_name():
@@ -269,7 +281,7 @@ def test_reference_cousin_package_different_name():
     )
 
     assert imports == {"from ...cousin import package2 as __cousin_package2__"}
-    assert name == "__cousin_package2__.Message"
+    assert name == '"__cousin_package2__.Message"'
 
 
 def test_reference_cousin_package_same_name():
@@ -279,7 +291,7 @@ def test_reference_cousin_package_same_name():
     )
 
     assert imports == {"from ...cousin import package as __cousin_package__"}
-    assert name == "__cousin_package__.Message"
+    assert name == '"__cousin_package__.Message"'
 
 
 def test_reference_far_cousin_package():
@@ -289,7 +301,7 @@ def test_reference_far_cousin_package():
     )
 
     assert imports == {"from ...b import c as __b_c__"}
-    assert name == "__b_c__.Message"
+    assert name == '"__b_c__.Message"'
 
 
 def test_reference_far_far_cousin_package():
@@ -299,7 +311,7 @@ def test_reference_far_far_cousin_package():
     )
 
     assert imports == {"from ....b.c import d as ___b_c_d__"}
-    assert name == "___b_c_d__.Message"
+    assert name == '"___b_c_d__.Message"'
 
 
 @pytest.mark.parametrize(

docs/upgrading.md

@@ -0,0 +1,16 @@
+# Upgrade Guide
+
+## [1.2.5] to [2.0.0b1]
+
+### Updated package structures
+
+Generated code now strictly follows the *package structure* of the `.proto` files.
+Consequently `.proto` files without a package will be combined in a single `__init__.py` file.
+To avoid overwriting existing `__init__.py` files, its best to compile into a dedicated subdirectory.
+
+Upgrading:
+
+- Remove your previously compiled `.py` files.
+- Create a new *empty* directory, e.g. `generated` or `lib/generated/proto` etcetera.
+- Regenerate your python files into this directory
+- Update import statements, e.g. `import ExampleMessage from generated`

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "betterproto"
-version = "1.2.5"
+version = "2.0.0b1"
 description = "A better Protobuf / gRPC generator & library"
 authors = ["Daniel G. Taylor <danielgtaylor@gmail.com>"]
 readme = "README.md"