refactor: make in_transaction default True

Added Documentation in Russian Language

CHANGELOG.md

@@ -2,6 +2,19 @@
 
 ## 0.7
 
+### 0.7.2
+
+- Support virtual fields.
+- Fix modify multiple times. (#279)
+- Added `-i` and `--in-transaction` options to `aerich migrate` command. (#296)
+- Fix generates two semicolons in a row. (#301)
+
+### 0.7.1
+
+- Fix syntax error with python3.8.10. (#265)
+- Fix sql generate error. (#263)
+- Fix initialize an empty database. (#267)
+
 ### 0.7.1rc1
 
 - Fix postgres sql error (#263)

Makefile

@@ -20,7 +20,7 @@ style: deps
 
 check: deps
 	@black --check $(black_opts) $(checkfiles) || (echo "Please run 'make style' to auto-fix style issues" && false)
-	@pflake8 $(checkfiles)
+	@ruff $(checkfiles)
 
 test: deps
 	$(py_warn) TEST_DB=sqlite://:memory: py.test

README.md

@@ -5,6 +5,8 @@
 [![image](https://github.com/tortoise/aerich/workflows/pypi/badge.svg)](https://github.com/tortoise/aerich/actions?query=workflow:pypi)
 [![image](https://github.com/tortoise/aerich/workflows/ci/badge.svg)](https://github.com/tortoise/aerich/actions?query=workflow:ci)
 
+English | [Русский](./README_RU.md)
+
 ## Introduction
 
 Aerich is a database migrations tool for TortoiseORM, which is like alembic for SQLAlchemy, or like Django ORM with

README_RU.md

@@ -0,0 +1,274 @@
+# Aerich
+ 
+[![image](https://img.shields.io/pypi/v/aerich.svg?style=flat)](https://pypi.python.org/pypi/aerich)
+[![image](https://img.shields.io/github/license/tortoise/aerich)](https://github.com/tortoise/aerich)
+[![image](https://github.com/tortoise/aerich/workflows/pypi/badge.svg)](https://github.com/tortoise/aerich/actions?query=workflow:pypi)
+[![image](https://github.com/tortoise/aerich/workflows/ci/badge.svg)](https://github.com/tortoise/aerich/actions?query=workflow:ci)
+ 
+[English](./README.md) | Русский
+
+## Введение
+ 
+Aerich - это инструмент для миграции базы данных для TortoiseORM, который аналогичен Alembic для SQLAlchemy или встроенному решению миграций в Django ORM.
+ 
+## Установка
+ 
+Просто установите из pypi:
+ 
+```shell
+pip install aerich
+```
+ 
+## Быстрый старт
+ 
+```shell
+> aerich -h
+ 
+Usage: aerich [OPTIONS] COMMAND [ARGS]...
+ 
+Options:
+  -V, --version      Show the version and exit.
+  -c, --config TEXT  Config file.  [default: pyproject.toml]
+  --app TEXT         Tortoise-ORM app name.
+  -h, --help         Show this message and exit.
+ 
+Commands:
+  downgrade  Downgrade to specified version.
+  heads      Show current available heads in migrate location.
+  history    List all migrate items.
+  init       Init config file and generate root migrate location.
+  init-db    Generate schema and generate app migrate location.
+  inspectdb  Introspects the database tables to standard output as...
+  migrate    Generate migrate changes file.
+  upgrade    Upgrade to specified version.
+```
+ 
+## Использование
+ 
+Сначала вам нужно добавить aerich.models в конфигурацию вашего Tortoise-ORM. Пример:
+ 
+```python
+TORTOISE_ORM = {
+    "connections": {"default": "mysql://root:123456@127.0.0.1:3306/test"},
+    "apps": {
+        "models": {
+            "models": ["tests.models", "aerich.models"],
+            "default_connection": "default",
+        },
+    },
+}
+```
+ 
+### Инициализация
+ 
+```shell
+> aerich init -h
+ 
+Usage: aerich init [OPTIONS]
+ 
+  Init config file and generate root migrate location.
+ 
+Options:
+  -t, --tortoise-orm TEXT  Tortoise-ORM config module dict variable, like
+                           settings.TORTOISE_ORM.  [required]
+  --location TEXT          Migrate store location.  [default: ./migrations]
+  -s, --src_folder TEXT    Folder of the source, relative to the project root.
+  -h, --help               Show this message and exit.
+```
+ 
+Инициализируйте файл конфигурации и задайте местоположение миграций:
+ 
+```shell
+> aerich init -t tests.backends.mysql.TORTOISE_ORM
+ 
+Success create migrate location ./migrations
+Success write config to pyproject.toml
+```
+ 
+### Инициализация базы данных
+ 
+```shell
+> aerich init-db
+ 
+Success create app migrate location ./migrations/models
+Success generate schema for app "models"
+```
+ 
+Если ваше приложение Tortoise-ORM не является приложением по умолчанию с именем models, вы должны указать правильное имя приложения с помощью параметра --app, например: aerich --app other_models init-db.
+ 
+### Обновление моделей и создание миграции
+ 
+```shell
+> aerich migrate --name drop_column
+ 
+Success migrate 1_202029051520102929_drop_column.py
+```
+ 
+Формат имени файла миграции следующий: `{версия}_{дата_и_время}_{имя|обновление}.py`.
+ 
+Если aerich предполагает, что вы переименовываете столбец, он спросит: 
+Переименовать `{старый_столбец} в {новый_столбец} [True]`. Вы можете выбрать `True`,
+чтобы переименовать столбец без удаления столбца, или выбрать `False`, чтобы удалить столбец,
+а затем создать новый. Обратите внимание, что последний вариант может привести к потере данных.
+ 
+ 
+### Обновление до последней версии
+ 
+```shell
+> aerich upgrade
+ 
+Success upgrade 1_202029051520102929_drop_column.py
+```
+ 
+Теперь ваша база данных обновлена до последней версии.
+ 
+### Откат до указанной версии
+ 
+```shell
+> aerich downgrade -h
+ 
+Usage: aerich downgrade [OPTIONS]
+ 
+  Downgrade to specified version.
+ 
+Options:
+  -v, --version INTEGER  Specified version, default to last.  [default: -1]
+  -d, --delete           Delete version files at the same time.  [default:
+                         False]
+ 
+  --yes                  Confirm the action without prompting.
+  -h, --help             Show this message and exit.
+```
+ 
+```shell
+> aerich downgrade
+ 
+Success downgrade 1_202029051520102929_drop_column.py
+```
+ 
+Теперь ваша база данных откатилась до указанной версии.
+ 
+### Показать историю
+ 
+```shell
+> aerich history
+ 
+1_202029051520102929_drop_column.py
+```
+ 
+### Чтобы узнать, какие миграции должны быть применены, можно использовать команду:
+ 
+```shell
+> aerich heads
+ 
+1_202029051520102929_drop_column.py
+```
+ 
+### Осмотр таблиц базы данных для модели TortoiseORM
+ 
+В настоящее время inspectdb поддерживает MySQL, Postgres и SQLite.
+ 
+```shell
+Usage: aerich inspectdb [OPTIONS]
+ 
+  Introspects the database tables to standard output as TortoiseORM model.
+ 
+Options:
+  -t, --table TEXT  Which tables to inspect.
+  -h, --help        Show this message and exit.
+```
+ 
+Посмотреть все таблицы и вывести их на консоль:
+ 
+```shell
+aerich --app models inspectdb
+```
+ 
+Осмотреть указанную таблицу в приложении по умолчанию и перенаправить в models.py:
+ 
+```shell
+aerich inspectdb -t user > models.py
+```
+ 
+Например, ваша таблица выглядит следующим образом:
+ 
+```sql
+CREATE TABLE `test`
+(
+    `id`       int            NOT NULL AUTO_INCREMENT,
+    `decimal`  decimal(10, 2) NOT NULL,
+    `date`     date                                    DEFAULT NULL,
+    `datetime` datetime       NOT NULL                 DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+    `time`     time                                    DEFAULT NULL,
+    `float`    float                                   DEFAULT NULL,
+    `string`   varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL,
+    `tinyint`  tinyint                                 DEFAULT NULL,
+    PRIMARY KEY (`id`),
+    KEY `asyncmy_string_index` (`string`)
+) ENGINE = InnoDB
+  DEFAULT CHARSET = utf8mb4
+  COLLATE = utf8mb4_general_ci
+```
+ 
+Теперь выполните команду aerich inspectdb -t test, чтобы увидеть сгенерированную модель:
+ 
+```python
+from tortoise import Model, fields
+ 
+ 
+class Test(Model):
+    date = fields.DateField(null=True, )
+    datetime = fields.DatetimeField(auto_now=True, )
+    decimal = fields.DecimalField(max_digits=10, decimal_places=2, )
+    float = fields.FloatField(null=True, )
+    id = fields.IntField(pk=True, )
+    string = fields.CharField(max_length=200, null=True, )
+    time = fields.TimeField(null=True, )
+    tinyint = fields.BooleanField(null=True, )
+```
+ 
+Обратите внимание, что эта команда имеет ограничения и не может автоматически определить некоторые поля, такие как `IntEnumField`, `ForeignKeyField` и другие.
+ 
+### Несколько баз данных
+ 
+```python
+tortoise_orm = {
+    "connections": {
+        "default": expand_db_url(db_url, True),
+        "second": expand_db_url(db_url_second, True),
+    },
+    "apps": {
+        "models": {"models": ["tests.models", "aerich.models"], "default_connection": "default"},
+        "models_second": {"models": ["tests.models_second"], "default_connection": "second", },
+    },
+}
+```
+ 
+Вам нужно указать `aerich.models` только в одном приложении и должны указывать `--app` при запуске команды `aerich migrate` и т.д.
+ 
+## Восстановление рабочего процесса aerich
+ 
+В некоторых случаях, например, при возникновении проблем после обновления `aerich`, вы не можете запустить `aerich migrate` или `aerich upgrade`. В таком случае вы можете выполнить следующие шаги:
+ 
+1. удалите таблицы `aerich`.
+2. удалите директорию `migrations/{app}`.
+3. rerun `aerich init-db`.
+ 
+Обратите внимание, что эти действия безопасны, и вы можете использовать их для сброса миграций, если у вас слишком много файлов миграции.
+ 
+## Использование aerich в приложении
+ 
+Вы можете использовать `aerich` вне командной строки, используя класс `Command`.
+ 
+```python
+from aerich import Command
+ 
+command = Command(tortoise_config=config, app='models')
+await command.init()
+await command.migrate('test')
+```
+ 
+## Лицензия
+ 
+Этот проект лицензирован в соответствии с лицензией
+[Apache-2.0](https://github.com/long2ice/aerich/blob/master/LICENSE) Лицензия.

aerich/__init__.py

@@ -36,7 +36,18 @@ class Command:
     async def init(self):
         await Migrate.init(self.tortoise_config, self.app, self.location)
 
-    async def upgrade(self):
+    async def _upgrade(self, conn, version_file):
+        file_path = Path(Migrate.migrate_location, version_file)
+        m = import_py_file(file_path)
+        upgrade = getattr(m, "upgrade")
+        await conn.execute_script(await upgrade(conn))
+        await Aerich.create(
+            version=version_file,
+            app=self.app,
+            content=get_models_describe(self.app),
+        )
+
+    async def upgrade(self, run_in_transaction: bool = True):
         migrated = []
         for version_file in Migrate.get_all_version_files():
             try:
@@ -44,18 +55,13 @@ class Command:
             except OperationalError:
                 exists = False
             if not exists:
-                async with in_transaction(
+                app_conn_name = get_app_connection_name(self.tortoise_config, self.app)
-                    get_app_connection_name(self.tortoise_config, self.app)
+                if run_in_transaction:
-                ) as conn:
+                    async with in_transaction(app_conn_name) as conn:
-                    file_path = Path(Migrate.migrate_location, version_file)
+                        await self._upgrade(conn, version_file)
-                    m = import_py_file(file_path)
+                else:
-                    upgrade = getattr(m, "upgrade")
+                    app_conn = get_app_connection(self.tortoise_config, self.app)
-                    await upgrade(conn)
+                    await self._upgrade(app_conn, version_file)
-                    await Aerich.create(
-                        version=version_file,
-                        app=self.app,
-                        content=get_models_describe(self.app),
-                    )
                 migrated.append(version_file)
         return migrated
 
@@ -80,10 +86,11 @@ class Command:
             ) as conn:
                 file_path = Path(Migrate.migrate_location, file)
                 m = import_py_file(file_path)
-                downgrade = getattr(m, "downgrade", None)
+                downgrade = getattr(m, "downgrade")
-                if not downgrade:
+                downgrade_sql = await downgrade(conn)
+                if not downgrade_sql.strip():
                     raise DowngradeError("No downgrade items found")
-                await downgrade(conn)
+                await conn.execute_script(downgrade_sql)
                 await version.delete()
                 if delete:
                     os.unlink(file_path)
@@ -138,6 +145,6 @@ class Command:
             content=get_models_describe(app),
         )
         version_file = Path(dirname, version)
-        content = MIGRATE_TEMPLATE.format(upgrade_sql=f'"""{schema}"""', downgrade_sql="")
+        content = MIGRATE_TEMPLATE.format(upgrade_sql=schema, downgrade_sql="")
         with open(version_file, "w", encoding="utf-8") as f:
             f.write(content)

aerich/cli.py

@@ -30,7 +30,7 @@ def coro(f):
         try:
             loop.run_until_complete(f(*args, **kwargs))
         finally:
-            if f.__name__ not in ["cli", "init_db", "init"]:
+            if f.__name__ not in ["cli", "init"]:
                 loop.run_until_complete(Tortoise.close_connections())
 
     return wrapper
@@ -90,11 +90,18 @@ async def migrate(ctx: Context, name):
 
 
 @cli.command(help="Upgrade to specified version.")
+@click.option(
+    "--in-transaction",
+    "-i",
+    default=True,
+    type=bool,
+    help="Make migrations in transaction or not. Can be helpful for large migrations or creating concurrent indexes.",
+)
 @click.pass_context
 @coro
-async def upgrade(ctx: Context):
+async def upgrade(ctx: Context, in_transaction: bool):
     command = ctx.obj["command"]
-    migrated = await command.upgrade()
+    migrated = await command.upgrade(run_in_transaction=in_transaction)
     if not migrated:
         click.secho("No upgrade items found", fg=Color.yellow)
     else:

aerich/ddl/__init__.py

@@ -40,7 +40,9 @@ class BaseDDL:
         self.schema_generator = self.schema_generator_cls(client)
 
     def create_table(self, model: "Type[Model]"):
-        return self.schema_generator._get_table_sql(model, True)["table_creation_string"]
+        return self.schema_generator._get_table_sql(model, True)["table_creation_string"].rstrip(
+            ";"
+        )
 
     def drop_table(self, table_name: str):
         return self._DROP_TABLE_TEMPLATE.format(table_name=table_name)

aerich/migrate.py

@@ -15,21 +15,17 @@ from aerich.ddl import BaseDDL
 from aerich.models import MAX_VERSION_LENGTH, Aerich
 from aerich.utils import get_app_connection, get_models_describe, is_default_function
 
-MIGRATE_TEMPLATE = """from typing import List
+MIGRATE_TEMPLATE = """from tortoise import BaseDBAsyncClient
-
-from tortoise import BaseDBAsyncClient
 
 
-async def upgrade(db: BaseDBAsyncClient) -> List[str]:
+async def upgrade(db: BaseDBAsyncClient) -> str:
-    return [
+    return \"\"\"
-        {upgrade_sql}
+        {upgrade_sql}\"\"\"
-    ]
 
 
-async def downgrade(db: BaseDBAsyncClient) -> List[str]:
+async def downgrade(db: BaseDBAsyncClient) -> str:
-    return [
+    return \"\"\"
-        {downgrade_sql}
+        {downgrade_sql}\"\"\"
-    ]
 """
 
 
@@ -52,7 +48,7 @@ class Migrate:
     _db_version: Optional[str] = None
 
     @classmethod
-    def get_all_version_files(cls) -> list[str]:
+    def get_all_version_files(cls) -> List[str]:
         return sorted(
             filter(lambda x: x.endswith("py"), os.listdir(cls.migrate_location)),
             key=lambda x: int(x.split("_")[0]),
@@ -125,9 +121,10 @@ class Migrate:
 
         version_file = Path(cls.migrate_location, version)
         content = MIGRATE_TEMPLATE.format(
-            upgrade_sql=",\n        ".join(map(lambda x: f"'{x}'", cls.upgrade_operators)),
+            upgrade_sql=";\n        ".join(cls.upgrade_operators) + ";",
-            downgrade_sql=",\n        ".join(map(lambda x: f"'{x}'", cls.downgrade_operators)),
+            downgrade_sql=";\n        ".join(cls.downgrade_operators) + ";",
         )
+
         with open(version_file, "w", encoding="utf-8") as f:
             f.write(content)
         return version
@@ -159,6 +156,7 @@ class Migrate:
         :param fk_m2m_index:
         :return:
         """
+        operator = operator.rstrip(";")
         if upgrade:
             if fk_m2m_index:
                 cls._upgrade_fk_m2m_index_operators.append(operator)
@@ -284,8 +282,18 @@ class Migrate:
                 # remove indexes
                 for index in old_indexes.difference(new_indexes):
                     cls._add_operator(cls._drop_index(model, index, False), upgrade, True)
-                old_data_fields = old_model_describe.get("data_fields")
+                old_data_fields = list(
-                new_data_fields = new_model_describe.get("data_fields")
+                    filter(
+                        lambda x: x.get("db_field_types") is not None,
+                        old_model_describe.get("data_fields"),
+                    )
+                )
+                new_data_fields = list(
+                    filter(
+                        lambda x: x.get("db_field_types") is not None,
+                        new_model_describe.get("data_fields"),
+                    )
+                )
 
                 old_data_fields_name = list(map(lambda x: x.get("name"), old_data_fields))
                 new_data_fields_name = list(map(lambda x: x.get("name"), new_data_fields))
@@ -441,6 +449,7 @@ class Migrate:
                         filter(lambda x: x.get("name") == field_name, new_data_fields)
                     )
                     changes = diff(old_data_field, new_data_field)
+                    modified = False
                     for change in changes:
                         _, option, old_new = change
                         if option == "indexed":
@@ -478,11 +487,14 @@ class Migrate:
                             # change nullable
                             cls._add_operator(cls._alter_null(model, new_data_field), upgrade)
                         else:
+                            if modified:
+                                continue
                             # modify column
                             cls._add_operator(
                                 cls._modify_field(model, new_data_field),
                                 upgrade,
                             )
+                            modified = True
 
         for old_model in old_models:
             if old_model not in new_models.keys():

aerich/version.py

@@ -1 +1 @@
-__version__ = "0.7.1rc1"
+__version__ = "0.7.2"

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "aerich"
-version = "0.7.1rc1"
+version = "0.7.2"
 description = "A database migrations tool for Tortoise ORM."
 authors = ["long2ice <long2ice@gmail.com>"]
 license = "Apache-2.0"
@@ -19,13 +19,13 @@ python = "^3.7"
 tortoise-orm = "*"
 click = "*"
 asyncpg = { version = "*", optional = true }
-asyncmy = { version = "*", optional = true }
+asyncmy = { version = "^0.2.8rc1", optional = true, allow-prereleases = true }
 pydantic = "*"
 dictdiffer = "*"
 tomlkit = "*"
 
 [tool.poetry.dev-dependencies]
-flake8 = "*"
+ruff = "*"
 isort = "*"
 black = "*"
 pytest = "*"
@@ -34,7 +34,6 @@ pytest-asyncio = "*"
 bandit = "*"
 pytest-mock = "*"
 cryptography = "*"
-pyproject-flake8 = "*"
 
 [tool.poetry.extras]
 asyncmy = ["asyncmy"]
@@ -63,5 +62,5 @@ asyncio_mode = 'auto'
 pretty = true
 ignore_missing_imports = true
 
-[tool.flake8]
+[tool.ruff]
-ignore = 'E501,W503,E203'
+ignore = ['E501']

tests/test_ddl.py

@@ -17,7 +17,7 @@ def test_create_table():
     `created_at` DATETIME(6) NOT NULL  DEFAULT CURRENT_TIMESTAMP(6),
     `user_id` INT NOT NULL COMMENT 'User',
     CONSTRAINT `fk_category_user_e2e3874c` FOREIGN KEY (`user_id`) REFERENCES `user` (`id`) ON DELETE CASCADE
-) CHARACTER SET utf8mb4;"""
+) CHARACTER SET utf8mb4"""
         )
 
     elif isinstance(Migrate.ddl, SqliteDDL):
@@ -29,7 +29,7 @@ def test_create_table():
     "name" VARCHAR(200),
     "created_at" TIMESTAMP NOT NULL  DEFAULT CURRENT_TIMESTAMP,
     "user_id" INT NOT NULL REFERENCES "user" ("id") ON DELETE CASCADE /* User */
-);"""
+)"""
         )
 
     elif isinstance(Migrate.ddl, PostgresDDL):
@@ -42,7 +42,7 @@ def test_create_table():
     "created_at" TIMESTAMPTZ NOT NULL  DEFAULT CURRENT_TIMESTAMP,
     "user_id" INT NOT NULL REFERENCES "user" ("id") ON DELETE CASCADE
 );
-COMMENT ON COLUMN "category"."user_id" IS 'User';"""
+COMMENT ON COLUMN "category"."user_id" IS 'User'"""
         )
 
 

tests/test_migrate.py

@@ -829,7 +829,7 @@ def test_migrate(mocker: MockerFixture):
             "ALTER TABLE `user` MODIFY COLUMN `longitude` DECIMAL(10,8) NOT NULL",
             "ALTER TABLE `user` ADD UNIQUE INDEX `uid_user_usernam_9987ab` (`username`)",
             "CREATE TABLE `email_user` (\n    `email_id` INT NOT NULL REFERENCES `email` (`email_id`) ON DELETE CASCADE,\n    `user_id` INT NOT NULL REFERENCES `user` (`id`) ON DELETE CASCADE\n) CHARACTER SET utf8mb4",
-            "CREATE TABLE IF NOT EXISTS `newmodel` (\n    `id` INT NOT NULL PRIMARY KEY AUTO_INCREMENT,\n    `name` VARCHAR(50) NOT NULL\n) CHARACTER SET utf8mb4;",
+            "CREATE TABLE IF NOT EXISTS `newmodel` (\n    `id` INT NOT NULL PRIMARY KEY AUTO_INCREMENT,\n    `name` VARCHAR(50) NOT NULL\n) CHARACTER SET utf8mb4",
             "ALTER TABLE `category` MODIFY COLUMN `created_at` DATETIME(6) NOT NULL  DEFAULT CURRENT_TIMESTAMP(6)",
             "ALTER TABLE `product` MODIFY COLUMN `body` LONGTEXT NOT NULL",
             "ALTER TABLE `email` MODIFY COLUMN `is_primary` BOOL NOT NULL  DEFAULT 0",
@@ -901,7 +901,7 @@ def test_migrate(mocker: MockerFixture):
             'CREATE INDEX "idx_product_name_869427" ON "product" ("name", "type_db_alias")',
             'CREATE INDEX "idx_email_email_4a1a33" ON "email" ("email")',
             'CREATE TABLE "email_user" (\n    "email_id" INT NOT NULL REFERENCES "email" ("email_id") ON DELETE CASCADE,\n    "user_id" INT NOT NULL REFERENCES "user" ("id") ON DELETE CASCADE\n)',
-            'CREATE TABLE IF NOT EXISTS "newmodel" (\n    "id" SERIAL NOT NULL PRIMARY KEY,\n    "name" VARCHAR(50) NOT NULL\n);\nCOMMENT ON COLUMN "config"."user_id" IS \'User\';',
+            'CREATE TABLE IF NOT EXISTS "newmodel" (\n    "id" SERIAL NOT NULL PRIMARY KEY,\n    "name" VARCHAR(50) NOT NULL\n);\nCOMMENT ON COLUMN "config"."user_id" IS \'User\'',
             'CREATE UNIQUE INDEX "uid_product_name_869427" ON "product" ("name", "type_db_alias")',
             'CREATE UNIQUE INDEX "uid_user_usernam_9987ab" ON "user" ("username")',
         }