21 Commits

Author SHA1 Message Date
long2ice
467406ed20 Merge pull request #303 from karichevi/dev
Added Documentation in Russian Language
2023-05-23 11:35:09 +08:00
karichevi
484b5900ce Added Documentation in Russian Language 2023-05-22 14:30:36 +03:00
long2ice
b8b6df0b65 chore: update deps 2023-05-12 15:27:50 +08:00
long2ice
f0bc3126e9 fix: generates two semicolons in a row. (#301) 2023-05-12 14:52:17 +08:00
long2ice
dbc0d9e7ef Merge pull request #296 from evstratbg/migrate-transaction
add in-transaction for upgrade
2023-05-05 23:12:08 +08:00
Bogdan
818dd29991 fix styles 2023-05-05 17:29:34 +04:00
Bogdan
e199e03b53 added -i param 2023-05-03 19:48:54 +04:00
Bogdan
d79dc25ee8 enriched changelog 2023-05-03 19:48:25 +04:00
Bogdan
c6d51a4dcf bump version 2023-05-03 19:48:15 +04:00
Bogdan
241b30a710 add in-transaction for upgrade 2023-04-03 20:55:12 +04:00
long2ice
8cf50c58d7 test: fix ci 2023-01-27 15:17:41 +08:00
long2ice
1c9b65cc37 fix: modify multiple times. (#279) 2023-01-27 13:49:07 +08:00
long2ice
3fbf9febfb Merge pull request #281 from CortexPE/patch-1
Fix #280 by removing trailing semicolon
2022-12-20 14:28:51 +08:00
marshall
7b6545d4e1 Fix #280 by removing trailing semicolon 2022-12-20 04:28:40 +08:00
long2ice
52b50a2161 feat: support virtual fields 2022-11-18 23:23:04 +08:00
long2ice
90943a473c Merge pull request #269 from jtraub/dev
Close connections in the init_db wrapper
2022-09-29 08:39:33 +08:00
Konstantin Mikhailov
d7ecd97e88 Close connections in the init_db wrapper 2022-09-29 08:09:59 +10:00
long2ice
20aebc4413 chore: update version 2022-09-27 22:44:12 +08:00
long2ice
f8e1f9ff44 fix: initialize an empty database. (#267) 2022-09-27 22:42:54 +08:00
long2ice
ab31445fb2 fix: test 2022-09-26 18:39:48 +08:00
long2ice
28d19a4b7b - Fix syntax error with python3.8.10. (#265)
- Fix sql generate error. (#263)
2022-09-26 18:36:57 +08:00
13 changed files with 1246 additions and 693 deletions

View File

@@ -2,6 +2,19 @@
## 0.7
### 0.7.2
- Support virtual fields.
- Fix modify multiple times. (#279)
- Added `-i` and `--in-transaction` options to `aerich migrate` command. (#296)
- Fix generates two semicolons in a row. (#301)
### 0.7.1
- Fix syntax error with python3.8.10. (#265)
- Fix sql generate error. (#263)
- Fix initialize an empty database. (#267)
### 0.7.1rc1
- Fix postgres sql error (#263)

View File

@@ -20,7 +20,7 @@ style: deps
check: deps
@black --check $(black_opts) $(checkfiles) || (echo "Please run 'make style' to auto-fix style issues" && false)
@pflake8 $(checkfiles)
@ruff $(checkfiles)
test: deps
$(py_warn) TEST_DB=sqlite://:memory: py.test

View File

@@ -5,6 +5,8 @@
[![image](https://github.com/tortoise/aerich/workflows/pypi/badge.svg)](https://github.com/tortoise/aerich/actions?query=workflow:pypi)
[![image](https://github.com/tortoise/aerich/workflows/ci/badge.svg)](https://github.com/tortoise/aerich/actions?query=workflow:ci)
English | [Русский](./README_RU.md)
## Introduction
Aerich is a database migrations tool for TortoiseORM, which is like alembic for SQLAlchemy, or like Django ORM with

274
README_RU.md Normal file
View File

@@ -0,0 +1,274 @@
# Aerich
[![image](https://img.shields.io/pypi/v/aerich.svg?style=flat)](https://pypi.python.org/pypi/aerich)
[![image](https://img.shields.io/github/license/tortoise/aerich)](https://github.com/tortoise/aerich)
[![image](https://github.com/tortoise/aerich/workflows/pypi/badge.svg)](https://github.com/tortoise/aerich/actions?query=workflow:pypi)
[![image](https://github.com/tortoise/aerich/workflows/ci/badge.svg)](https://github.com/tortoise/aerich/actions?query=workflow:ci)
[English](./README.md) | Русский
## Введение
Aerich - это инструмент для миграции базы данных для TortoiseORM, который аналогичен Alembic для SQLAlchemy или встроенному решению миграций в Django ORM.
## Установка
Просто установите из pypi:
```shell
pip install aerich
```
## Быстрый старт
```shell
> aerich -h
Usage: aerich [OPTIONS] COMMAND [ARGS]...
Options:
-V, --version Show the version and exit.
-c, --config TEXT Config file. [default: pyproject.toml]
--app TEXT Tortoise-ORM app name.
-h, --help Show this message and exit.
Commands:
downgrade Downgrade to specified version.
heads Show current available heads in migrate location.
history List all migrate items.
init Init config file and generate root migrate location.
init-db Generate schema and generate app migrate location.
inspectdb Introspects the database tables to standard output as...
migrate Generate migrate changes file.
upgrade Upgrade to specified version.
```
## Использование
Сначала вам нужно добавить aerich.models в конфигурацию вашего Tortoise-ORM. Пример:
```python
TORTOISE_ORM = {
"connections": {"default": "mysql://root:123456@127.0.0.1:3306/test"},
"apps": {
"models": {
"models": ["tests.models", "aerich.models"],
"default_connection": "default",
},
},
}
```
### Инициализация
```shell
> aerich init -h
Usage: aerich init [OPTIONS]
Init config file and generate root migrate location.
Options:
-t, --tortoise-orm TEXT Tortoise-ORM config module dict variable, like
settings.TORTOISE_ORM. [required]
--location TEXT Migrate store location. [default: ./migrations]
-s, --src_folder TEXT Folder of the source, relative to the project root.
-h, --help Show this message and exit.
```
Инициализируйте файл конфигурации и задайте местоположение миграций:
```shell
> aerich init -t tests.backends.mysql.TORTOISE_ORM
Success create migrate location ./migrations
Success write config to pyproject.toml
```
### Инициализация базы данных
```shell
> aerich init-db
Success create app migrate location ./migrations/models
Success generate schema for app "models"
```
Если ваше приложение Tortoise-ORM не является приложением по умолчанию с именем models, вы должны указать правильное имя приложения с помощью параметра --app, например: aerich --app other_models init-db.
### Обновление моделей и создание миграции
```shell
> aerich migrate --name drop_column
Success migrate 1_202029051520102929_drop_column.py
```
Формат имени файла миграции следующий: `{версия}_{дата_и_время}_{имя|обновление}.py`.
Если aerich предполагает, что вы переименовываете столбец, он спросит:
Переименовать `{старый_столбец} в {новый_столбец} [True]`. Вы можете выбрать `True`,
чтобы переименовать столбец без удаления столбца, или выбрать `False`, чтобы удалить столбец,
а затем создать новый. Обратите внимание, что последний вариант может привести к потере данных.
### Обновление до последней версии
```shell
> aerich upgrade
Success upgrade 1_202029051520102929_drop_column.py
```
Теперь ваша база данных обновлена до последней версии.
### Откат до указанной версии
```shell
> aerich downgrade -h
Usage: aerich downgrade [OPTIONS]
Downgrade to specified version.
Options:
-v, --version INTEGER Specified version, default to last. [default: -1]
-d, --delete Delete version files at the same time. [default:
False]
--yes Confirm the action without prompting.
-h, --help Show this message and exit.
```
```shell
> aerich downgrade
Success downgrade 1_202029051520102929_drop_column.py
```
Теперь ваша база данных откатилась до указанной версии.
### Показать историю
```shell
> aerich history
1_202029051520102929_drop_column.py
```
### Чтобы узнать, какие миграции должны быть применены, можно использовать команду:
```shell
> aerich heads
1_202029051520102929_drop_column.py
```
### Осмотр таблиц базы данных для модели TortoiseORM
В настоящее время inspectdb поддерживает MySQL, Postgres и SQLite.
```shell
Usage: aerich inspectdb [OPTIONS]
Introspects the database tables to standard output as TortoiseORM model.
Options:
-t, --table TEXT Which tables to inspect.
-h, --help Show this message and exit.
```
Посмотреть все таблицы и вывести их на консоль:
```shell
aerich --app models inspectdb
```
Осмотреть указанную таблицу в приложении по умолчанию и перенаправить в models.py:
```shell
aerich inspectdb -t user > models.py
```
Например, ваша таблица выглядит следующим образом:
```sql
CREATE TABLE `test`
(
`id` int NOT NULL AUTO_INCREMENT,
`decimal` decimal(10, 2) NOT NULL,
`date` date DEFAULT NULL,
`datetime` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`time` time DEFAULT NULL,
`float` float DEFAULT NULL,
`string` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL,
`tinyint` tinyint DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `asyncmy_string_index` (`string`)
) ENGINE = InnoDB
DEFAULT CHARSET = utf8mb4
COLLATE = utf8mb4_general_ci
```
Теперь выполните команду aerich inspectdb -t test, чтобы увидеть сгенерированную модель:
```python
from tortoise import Model, fields
class Test(Model):
date = fields.DateField(null=True, )
datetime = fields.DatetimeField(auto_now=True, )
decimal = fields.DecimalField(max_digits=10, decimal_places=2, )
float = fields.FloatField(null=True, )
id = fields.IntField(pk=True, )
string = fields.CharField(max_length=200, null=True, )
time = fields.TimeField(null=True, )
tinyint = fields.BooleanField(null=True, )
```
Обратите внимание, что эта команда имеет ограничения и не может автоматически определить некоторые поля, такие как `IntEnumField`, `ForeignKeyField` и другие.
### Несколько баз данных
```python
tortoise_orm = {
"connections": {
"default": expand_db_url(db_url, True),
"second": expand_db_url(db_url_second, True),
},
"apps": {
"models": {"models": ["tests.models", "aerich.models"], "default_connection": "default"},
"models_second": {"models": ["tests.models_second"], "default_connection": "second", },
},
}
```
Вам нужно указать `aerich.models` только в одном приложении и должны указывать `--app` при запуске команды `aerich migrate` и т.д.
## Восстановление рабочего процесса aerich
В некоторых случаях, например, при возникновении проблем после обновления `aerich`, вы не можете запустить `aerich migrate` или `aerich upgrade`. В таком случае вы можете выполнить следующие шаги:
1. удалите таблицы `aerich`.
2. удалите директорию `migrations/{app}`.
3. rerun `aerich init-db`.
Обратите внимание, что эти действия безопасны, и вы можете использовать их для сброса миграций, если у вас слишком много файлов миграции.
## Использование aerich в приложении
Вы можете использовать `aerich` вне командной строки, используя класс `Command`.
```python
from aerich import Command
command = Command(tortoise_config=config, app='models')
await command.init()
await command.migrate('test')
```
## Лицензия
Этот проект лицензирован в соответствии с лицензией
[Apache-2.0](https://github.com/long2ice/aerich/blob/master/LICENSE) Лицензия.

View File

@@ -36,7 +36,18 @@ class Command:
async def init(self):
await Migrate.init(self.tortoise_config, self.app, self.location)
async def upgrade(self):
async def _upgrade(self, conn, version_file):
file_path = Path(Migrate.migrate_location, version_file)
m = import_py_file(file_path)
upgrade = getattr(m, "upgrade")
await conn.execute_script(await upgrade(conn))
await Aerich.create(
version=version_file,
app=self.app,
content=get_models_describe(self.app),
)
async def upgrade(self, run_in_transaction: bool):
migrated = []
for version_file in Migrate.get_all_version_files():
try:
@@ -44,18 +55,13 @@ class Command:
except OperationalError:
exists = False
if not exists:
async with in_transaction(
get_app_connection_name(self.tortoise_config, self.app)
) as conn:
file_path = Path(Migrate.migrate_location, version_file)
m = import_py_file(file_path)
upgrade = getattr(m, "upgrade")
await upgrade(conn)
await Aerich.create(
version=version_file,
app=self.app,
content=get_models_describe(self.app),
)
app_conn_name = get_app_connection_name(self.tortoise_config, self.app)
if run_in_transaction:
async with in_transaction(app_conn_name) as conn:
await self._upgrade(conn, version_file)
else:
app_conn = get_app_connection(self.tortoise_config, self.app)
await self._upgrade(app_conn, version_file)
migrated.append(version_file)
return migrated
@@ -80,10 +86,11 @@ class Command:
) as conn:
file_path = Path(Migrate.migrate_location, file)
m = import_py_file(file_path)
downgrade = getattr(m, "downgrade", None)
if not downgrade:
downgrade = getattr(m, "downgrade")
downgrade_sql = await downgrade(conn)
if not downgrade_sql.strip():
raise DowngradeError("No downgrade items found")
await downgrade(conn)
await conn.execute_script(downgrade_sql)
await version.delete()
if delete:
os.unlink(file_path)
@@ -138,6 +145,6 @@ class Command:
content=get_models_describe(app),
)
version_file = Path(dirname, version)
content = MIGRATE_TEMPLATE.format(upgrade_sql=f'"""{schema}"""', downgrade_sql="")
content = MIGRATE_TEMPLATE.format(upgrade_sql=schema, downgrade_sql="")
with open(version_file, "w", encoding="utf-8") as f:
f.write(content)

View File

@@ -30,7 +30,7 @@ def coro(f):
try:
loop.run_until_complete(f(*args, **kwargs))
finally:
if f.__name__ not in ["cli", "init_db", "init"]:
if f.__name__ not in ["cli", "init"]:
loop.run_until_complete(Tortoise.close_connections())
return wrapper
@@ -90,11 +90,18 @@ async def migrate(ctx: Context, name):
@cli.command(help="Upgrade to specified version.")
@click.option(
"--in-transaction",
"-i",
default=True,
type=bool,
help="Make migrations in transaction or not. Can be helpful for large migrations or creating concurrent indexes.",
)
@click.pass_context
@coro
async def upgrade(ctx: Context):
async def upgrade(ctx: Context, in_transaction: bool):
command = ctx.obj["command"]
migrated = await command.upgrade()
migrated = await command.upgrade(run_in_transaction=in_transaction)
if not migrated:
click.secho("No upgrade items found", fg=Color.yellow)
else:

View File

@@ -40,7 +40,9 @@ class BaseDDL:
self.schema_generator = self.schema_generator_cls(client)
def create_table(self, model: "Type[Model]"):
return self.schema_generator._get_table_sql(model, True)["table_creation_string"]
return self.schema_generator._get_table_sql(model, True)["table_creation_string"].rstrip(
";"
)
def drop_table(self, table_name: str):
return self._DROP_TABLE_TEMPLATE.format(table_name=table_name)

View File

@@ -15,21 +15,17 @@ from aerich.ddl import BaseDDL
from aerich.models import MAX_VERSION_LENGTH, Aerich
from aerich.utils import get_app_connection, get_models_describe, is_default_function
MIGRATE_TEMPLATE = """from typing import List
from tortoise import BaseDBAsyncClient
MIGRATE_TEMPLATE = """from tortoise import BaseDBAsyncClient
async def upgrade(db: BaseDBAsyncClient) -> List[str]:
return [
{upgrade_sql}
]
async def upgrade(db: BaseDBAsyncClient) -> str:
return \"\"\"
{upgrade_sql}\"\"\"
async def downgrade(db: BaseDBAsyncClient) -> List[str]:
return [
{downgrade_sql}
]
async def downgrade(db: BaseDBAsyncClient) -> str:
return \"\"\"
{downgrade_sql}\"\"\"
"""
@@ -52,7 +48,7 @@ class Migrate:
_db_version: Optional[str] = None
@classmethod
def get_all_version_files(cls) -> list[str]:
def get_all_version_files(cls) -> List[str]:
return sorted(
filter(lambda x: x.endswith("py"), os.listdir(cls.migrate_location)),
key=lambda x: int(x.split("_")[0]),
@@ -125,9 +121,10 @@ class Migrate:
version_file = Path(cls.migrate_location, version)
content = MIGRATE_TEMPLATE.format(
upgrade_sql=",\n ".join(map(lambda x: f"'{x}'", cls.upgrade_operators)),
downgrade_sql=",\n ".join(map(lambda x: f"'{x}'", cls.downgrade_operators)),
upgrade_sql=";\n ".join(cls.upgrade_operators) + ";",
downgrade_sql=";\n ".join(cls.downgrade_operators) + ";",
)
with open(version_file, "w", encoding="utf-8") as f:
f.write(content)
return version
@@ -159,6 +156,7 @@ class Migrate:
:param fk_m2m_index:
:return:
"""
operator = operator.rstrip(";")
if upgrade:
if fk_m2m_index:
cls._upgrade_fk_m2m_index_operators.append(operator)
@@ -284,8 +282,18 @@ class Migrate:
# remove indexes
for index in old_indexes.difference(new_indexes):
cls._add_operator(cls._drop_index(model, index, False), upgrade, True)
old_data_fields = old_model_describe.get("data_fields")
new_data_fields = new_model_describe.get("data_fields")
old_data_fields = list(
filter(
lambda x: x.get("db_field_types") is not None,
old_model_describe.get("data_fields"),
)
)
new_data_fields = list(
filter(
lambda x: x.get("db_field_types") is not None,
new_model_describe.get("data_fields"),
)
)
old_data_fields_name = list(map(lambda x: x.get("name"), old_data_fields))
new_data_fields_name = list(map(lambda x: x.get("name"), new_data_fields))
@@ -441,6 +449,7 @@ class Migrate:
filter(lambda x: x.get("name") == field_name, new_data_fields)
)
changes = diff(old_data_field, new_data_field)
modified = False
for change in changes:
_, option, old_new = change
if option == "indexed":
@@ -478,11 +487,14 @@ class Migrate:
# change nullable
cls._add_operator(cls._alter_null(model, new_data_field), upgrade)
else:
if modified:
continue
# modify column
cls._add_operator(
cls._modify_field(model, new_data_field),
upgrade,
)
modified = True
for old_model in old_models:
if old_model not in new_models.keys():

View File

@@ -1 +1 @@
__version__ = "0.7.1rc1"
__version__ = "0.7.2"

1523
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,6 +1,6 @@
[tool.poetry]
name = "aerich"
version = "0.7.1rc1"
version = "0.7.2"
description = "A database migrations tool for Tortoise ORM."
authors = ["long2ice <long2ice@gmail.com>"]
license = "Apache-2.0"
@@ -19,13 +19,13 @@ python = "^3.7"
tortoise-orm = "*"
click = "*"
asyncpg = { version = "*", optional = true }
asyncmy = { version = "*", optional = true }
asyncmy = { version = "^0.2.8rc1", optional = true, allow-prereleases = true }
pydantic = "*"
dictdiffer = "*"
tomlkit = "*"
[tool.poetry.dev-dependencies]
flake8 = "*"
ruff = "*"
isort = "*"
black = "*"
pytest = "*"
@@ -34,7 +34,6 @@ pytest-asyncio = "*"
bandit = "*"
pytest-mock = "*"
cryptography = "*"
pyproject-flake8 = "*"
[tool.poetry.extras]
asyncmy = ["asyncmy"]
@@ -63,5 +62,5 @@ asyncio_mode = 'auto'
pretty = true
ignore_missing_imports = true
[tool.flake8]
ignore = 'E501,W503,E203'
[tool.ruff]
ignore = ['E501']

View File

@@ -17,7 +17,7 @@ def test_create_table():
`created_at` DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
`user_id` INT NOT NULL COMMENT 'User',
CONSTRAINT `fk_category_user_e2e3874c` FOREIGN KEY (`user_id`) REFERENCES `user` (`id`) ON DELETE CASCADE
) CHARACTER SET utf8mb4;"""
) CHARACTER SET utf8mb4"""
)
elif isinstance(Migrate.ddl, SqliteDDL):
@@ -29,7 +29,7 @@ def test_create_table():
"name" VARCHAR(200),
"created_at" TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
"user_id" INT NOT NULL REFERENCES "user" ("id") ON DELETE CASCADE /* User */
);"""
)"""
)
elif isinstance(Migrate.ddl, PostgresDDL):
@@ -42,7 +42,7 @@ def test_create_table():
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
"user_id" INT NOT NULL REFERENCES "user" ("id") ON DELETE CASCADE
);
COMMENT ON COLUMN "category"."user_id" IS 'User';"""
COMMENT ON COLUMN "category"."user_id" IS 'User'"""
)

View File

@@ -829,7 +829,7 @@ def test_migrate(mocker: MockerFixture):
"ALTER TABLE `user` MODIFY COLUMN `longitude` DECIMAL(10,8) NOT NULL",
"ALTER TABLE `user` ADD UNIQUE INDEX `uid_user_usernam_9987ab` (`username`)",
"CREATE TABLE `email_user` (\n `email_id` INT NOT NULL REFERENCES `email` (`email_id`) ON DELETE CASCADE,\n `user_id` INT NOT NULL REFERENCES `user` (`id`) ON DELETE CASCADE\n) CHARACTER SET utf8mb4",
"CREATE TABLE IF NOT EXISTS `newmodel` (\n `id` INT NOT NULL PRIMARY KEY AUTO_INCREMENT,\n `name` VARCHAR(50) NOT NULL\n) CHARACTER SET utf8mb4;",
"CREATE TABLE IF NOT EXISTS `newmodel` (\n `id` INT NOT NULL PRIMARY KEY AUTO_INCREMENT,\n `name` VARCHAR(50) NOT NULL\n) CHARACTER SET utf8mb4",
"ALTER TABLE `category` MODIFY COLUMN `created_at` DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6)",
"ALTER TABLE `product` MODIFY COLUMN `body` LONGTEXT NOT NULL",
"ALTER TABLE `email` MODIFY COLUMN `is_primary` BOOL NOT NULL DEFAULT 0",
@@ -901,7 +901,7 @@ def test_migrate(mocker: MockerFixture):
'CREATE INDEX "idx_product_name_869427" ON "product" ("name", "type_db_alias")',
'CREATE INDEX "idx_email_email_4a1a33" ON "email" ("email")',
'CREATE TABLE "email_user" (\n "email_id" INT NOT NULL REFERENCES "email" ("email_id") ON DELETE CASCADE,\n "user_id" INT NOT NULL REFERENCES "user" ("id") ON DELETE CASCADE\n)',
'CREATE TABLE IF NOT EXISTS "newmodel" (\n "id" SERIAL NOT NULL PRIMARY KEY,\n "name" VARCHAR(50) NOT NULL\n);\nCOMMENT ON COLUMN "config"."user_id" IS \'User\';',
'CREATE TABLE IF NOT EXISTS "newmodel" (\n "id" SERIAL NOT NULL PRIMARY KEY,\n "name" VARCHAR(50) NOT NULL\n);\nCOMMENT ON COLUMN "config"."user_id" IS \'User\'',
'CREATE UNIQUE INDEX "uid_product_name_869427" ON "product" ("name", "type_db_alias")',
'CREATE UNIQUE INDEX "uid_user_usernam_9987ab" ON "user" ("username")',
}