37 Commits

Author SHA1 Message Date
long2ice
c707f7ecb2 bug fix 2020-11-28 14:31:41 +08:00
long2ice
0bbc471e00 Fix sqlite stuck. (#90) 2020-11-26 23:38:57 +08:00
long2ice
fb6cc62047 update README and CHANGELOG 2020-11-23 16:44:16 +08:00
long2ice
e9ceaf471f Merge pull request #87 from ALexALed/remove-default-detections-for-callable
Remove callable detection for defaults
2020-11-23 16:41:30 +08:00
alexaled
85fc3b2aa2 Remove callable detection for defaults 2020-11-23 10:35:40 +02:00
long2ice
a677d506a9 Fix ci error 2020-11-19 10:41:52 +08:00
long2ice
9879004fee Add rename column support MySQL5 2020-11-19 10:11:52 +08:00
long2ice
5760fe2040 Merge pull request #83 from SakuraSound/fix-migrate-unlink
Catch OSError (if read-only file system)
2020-11-18 15:40:29 +08:00
Joir-dan Gumbs
b229c30558 Catch OSError (if read-only file system) 2020-11-17 23:28:00 -08:00
long2ice
5d2f1604c3 update github action poetry 2020-11-17 10:57:56 +08:00
long2ice
499c4e1c02 Fix black 2020-11-17 10:50:57 +08:00
long2ice
1463ee30bc update deps 2020-11-17 10:43:27 +08:00
long2ice
3b801932f5 Merge remote-tracking branch 'origin/dev' into dev 2020-11-17 10:36:14 +08:00
long2ice
c2eb4dc9e3 update poetry in github actions 2020-11-17 10:35:51 +08:00
long2ice
5927febd0c Delete .DS_Store 2020-11-17 10:10:32 +08:00
long2ice
a1c10ff330 exclude .DS_store 2020-11-17 10:09:37 +08:00
long2ice
f2013c931a Fix test error 2020-11-16 22:32:19 +08:00
long2ice
b21b954d32 Use .sql instead of .json to store version file. (#79) 2020-11-16 22:25:01 +08:00
long2ice
f5588a35c5 update deps 2020-11-12 21:27:58 +08:00
long2ice
f5dff84476 Fix encoding error. (#75) 2020-11-08 23:00:44 +08:00
long2ice
e399821116 update deps 2020-11-05 17:43:41 +08:00
long2ice
648f25a951 Compatible with models file in directory. (#70) 2020-10-30 19:51:46 +08:00
long2ice
fa73e132e2 remove .vscode 2020-10-30 16:45:12 +08:00
long2ice
1bac33cd33 add confirmation_option when downgrade 2020-10-30 16:39:14 +08:00
long2ice
4e76f12ccf update README.md 2020-10-28 17:12:23 +08:00
long2ice
724379700e Support multiple databases. (#68) 2020-10-28 17:02:02 +08:00
long2ice
bb929f2b55 update deps 2020-10-25 17:48:05 +08:00
long2ice
6339dc86a8 Fix migrate to new database error 2020-10-14 20:33:23 +08:00
long2ice
768747140a update changelog 2020-10-12 21:01:53 +08:00
long2ice
1fde3cd04e fix init KeyError (#61) 2020-10-12 20:59:13 +08:00
long2ice
d0ce545ff5 Fix first version error 2020-10-10 15:07:09 +08:00
long2ice
09b89ed7d0 update README 2020-10-09 15:41:37 +08:00
long2ice
86c8382593 update README 2020-10-09 15:34:48 +08:00
long2ice
48e3ff48a3 update README.md 2020-10-09 12:04:36 +08:00
long2ice
1bf6d45bb0 Merge pull request #60 from tortoise/refactoring
Refactoring migrate logic
2020-10-09 11:54:37 +08:00
long2ice
342f4cdd3b complete refactoring 2020-10-09 11:53:50 +08:00
long2ice
8cace21fde refactoring migrate logic 2020-10-09 00:05:22 +08:00
19 changed files with 831 additions and 497 deletions

View File

@@ -11,11 +11,14 @@ jobs:
- uses: actions/setup-python@v2
with:
python-version: '3.x'
- uses: dschep/install-poetry-action@v1.3
- name: Install and configure Poetry
uses: snok/install-poetry@v1.1.1
with:
virtualenvs-create: false
- name: Build dists
run: make build
- name: Pypi Publish
uses: pypa/gh-action-pypi-publish@master
with:
user: __token__
password: ${{ secrets.pypi_password }}
password: ${{ secrets.pypi_password }}

View File

@@ -1,5 +1,5 @@
name: test
on: [push, pull_request]
on: [ push, pull_request ]
jobs:
testall:
runs-on: ubuntu-latest
@@ -19,7 +19,10 @@ jobs:
- uses: actions/setup-python@v2
with:
python-version: '3.x'
- uses: dschep/install-poetry-action@v1.3
- name: Install and configure Poetry
uses: snok/install-poetry@v1.1.1
with:
virtualenvs-create: false
- name: CI
env:
MYSQL_PASS: root
@@ -28,4 +31,4 @@ jobs:
POSTGRES_PASS: 123456
POSTGRES_HOST: 127.0.0.1
POSTGRES_PORT: 5432
run: make ci
run: make ci

4
.gitignore vendored
View File

@@ -143,4 +143,6 @@ cython_debug/
.idea
migrations
aerich.ini
src
src
.vscode
.DS_Store

View File

@@ -1,5 +1,41 @@
# ChangeLog
## 0.4
### 0.4.0
- Use `.sql` instead of `.json` to store version file.
- Add `rename` column support MySQL5.
- Remove callable detection for defaults. (#87)
- Fix `sqlite` stuck. (#90)
## 0.3
### 0.3.3
- Fix encoding error. (#75)
- Support multiple databases. (#68)
- Compatible with models file in directory. (#70)
### 0.3.2
- Fix migrate to new database error. (#62)
### 0.3.1
- Fix first version error.
- Fix init error. (#61)
### 0.3.0
- Refactoring migrate logic, and this version is not compatible with previous version.
- Now there don't need `old_models.py` and it store in database.
- Upgrade steps:
1. Upgrade aerich version.
2. Drop aerich table in database.
3. Delete `migrations/{app}` folder and rerun `aerich init-db`.
4. Update model and `aerich migrate` normally.
## 0.2
### 0.2.5

View File

@@ -3,8 +3,10 @@ black_opts = -l 100 -t py38
py_warn = PYTHONDEVMODE=1
MYSQL_HOST ?= "127.0.0.1"
MYSQL_PORT ?= 3306
MYSQL_PASS ?= "123456"
POSTGRES_HOST ?= "127.0.0.1"
POSTGRES_PORT ?= 5432
POSTGRES_PASS ?= "123456"
help:
@echo "Aerich development makefile"
@@ -22,7 +24,7 @@ up:
@poetry update
deps:
@poetry install -E dbdrivers --no-root
@poetry install -E dbdrivers
style: deps
isort -src $(checkfiles)

View File

@@ -7,12 +7,10 @@
## Introduction
Tortoise-ORM is the best asyncio ORM now, but it lacks a database
migrations tool like alembic for SQLAlchemy, or Django ORM with it\'s
own migrations tool.
Aerich is a database migrations tool for Tortoise-ORM, which like alembic for SQLAlchemy, or Django ORM with it\'s
own migrations solution.
This project aim to be a best migrations tool for Tortoise-ORM and which
written by one of contributors of Tortoise-ORM.
**Important: You can only use absolutely import in your `models.py` to make `aerich` work.**
## Install
@@ -25,7 +23,7 @@ Just install from pypi:
## Quick Start
```shell
$ aerich -h
> aerich -h
Usage: aerich [OPTIONS] COMMAND [ARGS]...
@@ -37,7 +35,7 @@ Options:
-h, --help Show this message and exit.
Commands:
downgrade Downgrade to previous version.
downgrade Downgrade to specified version.
heads Show current available heads in migrate location.
history List all migrate items.
init Init config file and generate root migrate location.
@@ -66,7 +64,7 @@ TORTOISE_ORM = {
### Initialization
```shell
$ aerich init -h
> aerich init -h
Usage: aerich init [OPTIONS]
@@ -82,7 +80,7 @@ Options:
Init config file and location:
```shell
$ aerich init -t tests.backends.mysql.TORTOISE_ORM
> aerich init -t tests.backends.mysql.TORTOISE_ORM
Success create migrate location ./migrations
Success generate config file aerich.ini
@@ -91,7 +89,7 @@ Success generate config file aerich.ini
### Init db
```shell
$ aerich init-db
> aerich init-db
Success create app migrate location ./migrations/models
Success generate schema for app "models"
@@ -103,58 +101,86 @@ If your Tortoise-ORM app is not default `models`, you must specify
### Update models and make migrate
```shell
$ aerich migrate --name drop_column
> aerich migrate --name drop_column
Success migrate 1_202029051520102929_drop_column.json
Success migrate 1_202029051520102929_drop_column.sql
```
Format of migrate filename is
`{version_num}_{datetime}_{name|update}.json`.
`{version_num}_{datetime}_{name|update}.sql`.
And if `aerich` guess you are renaming a column, it will ask `Rename {old_column} to {new_column} [True]`, you can choice `True` to rename column without column drop, or choice `False` to drop column then create.
If you use `MySQL`, only MySQL8.0+ support `rename..to` syntax.
### Upgrade to latest version
```shell
$ aerich upgrade
> aerich upgrade
Success upgrade 1_202029051520102929_drop_column.json
Success upgrade 1_202029051520102929_drop_column.sql
```
Now your db is migrated to latest.
### Downgrade to previous version
### Downgrade to specified version
```shell
$ aerich downgrade
> aerich init -h
Success downgrade 1_202029051520102929_drop_column.json
Usage: aerich downgrade [OPTIONS]
Downgrade to specified version.
Options:
-v, --version INTEGER Specified version, default to last. [default: -1]
-d, --delete Delete version files at the same time. [default:
False]
--yes Confirm the action without prompting.
-h, --help Show this message and exit.
```
Now your db rollback to previous version.
```shell
> aerich downgrade
Success downgrade 1_202029051520102929_drop_column.sql
```
Now your db rollback to specified version.
### Show history
```shell
$ aerich history
> aerich history
1_202029051520102929_drop_column.json
1_202029051520102929_drop_column.sql
```
### Show heads to be migrated
```shell
$ aerich heads
> aerich heads
1_202029051520102929_drop_column.json
1_202029051520102929_drop_column.sql
```
## Support this project
### Multiple databases
- Just give a star!
- Donation.
```python
tortoise_orm = {
"connections": {
"default": expand_db_url(db_url, True),
"second": expand_db_url(db_url_second, True),
},
"apps": {
"models": {"models": ["tests.models", "aerich.models"], "default_connection": "default"},
"models_second": {"models": ["tests.models_second"], "default_connection": "second",},
},
}
```
You need only specify `aerich.models` in one app, and must specify `--app` when run `aerich migrate` and so on.
## Support this project
| AliPay | WeChatPay | PayPal |
| -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |

View File

@@ -1 +1 @@
__version__ = "0.2.5"
__version__ = "0.4.1"

View File

@@ -1,5 +1,4 @@
import asyncio
import json
import os
import sys
from configparser import ConfigParser
@@ -13,7 +12,13 @@ from tortoise.transactions import in_transaction
from tortoise.utils import get_schema_sql
from aerich.migrate import Migrate
from aerich.utils import get_app_connection, get_app_connection_name, get_tortoise_config
from aerich.utils import (
get_app_connection,
get_app_connection_name,
get_tortoise_config,
get_version_content_from_file,
write_version_file,
)
from . import __version__
from .enums import Color
@@ -26,7 +31,11 @@ def coro(f):
@wraps(f)
def wrapper(*args, **kwargs):
loop = asyncio.get_event_loop()
return loop.run_until_complete(f(*args, **kwargs))
ctx = args[0]
loop.run_until_complete(f(*args, **kwargs))
app = ctx.obj.get("app")
if app:
Migrate.remove_old_model_file(app, ctx.obj["location"])
return wrapper
@@ -34,7 +43,11 @@ def coro(f):
@click.group(context_settings={"help_option_names": ["-h", "--help"]})
@click.version_option(__version__, "-V", "--version")
@click.option(
"-c", "--config", default="aerich.ini", show_default=True, help="Config file.",
"-c",
"--config",
default="aerich.ini",
show_default=True,
help="Config file.",
)
@click.option("--app", required=False, help="Tortoise-ORM app name.")
@click.option(
@@ -62,12 +75,10 @@ async def cli(ctx: Context, config, app, name):
tortoise_config = get_tortoise_config(ctx, tortoise_orm)
app = app or list(tortoise_config.get("apps").keys())[0]
if "aerich.models" not in tortoise_config.get("apps").get(app).get("models"):
raise UsageError("Check your tortoise config and add aerich.models to it.", ctx=ctx)
ctx.obj["config"] = tortoise_config
ctx.obj["location"] = location
ctx.obj["app"] = app
Migrate.app = app
if invoked_subcommand != "init-db":
await Migrate.init_with_old_models(tortoise_config, app, location)
@@ -77,64 +88,92 @@ async def cli(ctx: Context, config, app, name):
@click.pass_context
@coro
async def migrate(ctx: Context, name):
config = ctx.obj["config"]
location = ctx.obj["location"]
app = ctx.obj["app"]
ret = await Migrate.migrate(name)
if not ret:
return click.secho("No changes detected", fg=Color.yellow)
Migrate.write_old_models(config, app, location)
click.secho(f"Success migrate {ret}", fg=Color.green)
@cli.command(help="Upgrade to latest version.")
@cli.command(help="Upgrade to specified version.")
@click.pass_context
@coro
async def upgrade(ctx: Context):
config = ctx.obj["config"]
app = ctx.obj["app"]
location = ctx.obj["location"]
migrated = False
for version in Migrate.get_all_version_files():
for version_file in Migrate.get_all_version_files():
try:
exists = await Aerich.exists(version=version, app=app)
exists = await Aerich.exists(version=version_file, app=app)
except OperationalError:
exists = False
if not exists:
async with in_transaction(get_app_connection_name(config, app)) as conn:
file_path = os.path.join(Migrate.migrate_location, version)
with open(file_path, "r", encoding="utf-8") as f:
content = json.load(f)
upgrade_query_list = content.get("upgrade")
for upgrade_query in upgrade_query_list:
await conn.execute_script(upgrade_query)
await Aerich.create(version=version, app=app)
click.secho(f"Success upgrade {version}", fg=Color.green)
file_path = os.path.join(Migrate.migrate_location, version_file)
content = get_version_content_from_file(file_path)
upgrade_query_list = content.get("upgrade")
for upgrade_query in upgrade_query_list:
await conn.execute_script(upgrade_query)
await Aerich.create(
version=version_file,
app=app,
content=Migrate.get_models_content(config, app, location),
)
click.secho(f"Success upgrade {version_file}", fg=Color.green)
migrated = True
if not migrated:
click.secho("No migrate items", fg=Color.yellow)
@cli.command(help="Downgrade to previous version.")
@cli.command(help="Downgrade to specified version.")
@click.option(
"-v",
"--version",
default=-1,
type=int,
show_default=True,
help="Specified version, default to last.",
)
@click.option(
"-d",
"--delete",
is_flag=True,
default=False,
show_default=True,
help="Delete version files at the same time.",
)
@click.pass_context
@click.confirmation_option(
prompt="Downgrade is dangerous, which maybe lose your data, are you sure?",
)
@coro
async def downgrade(ctx: Context):
async def downgrade(ctx: Context, version: int, delete: bool):
app = ctx.obj["app"]
config = ctx.obj["config"]
last_version = await Migrate.get_last_version()
if not last_version:
return click.secho("No last version found", fg=Color.yellow)
file = last_version.version
async with in_transaction(get_app_connection_name(config, app)) as conn:
file_path = os.path.join(Migrate.migrate_location, file)
with open(file_path, "r", encoding="utf-8") as f:
content = json.load(f)
if version == -1:
specified_version = await Migrate.get_last_version()
else:
specified_version = await Aerich.filter(app=app, version__startswith=f"{version}_").first()
if not specified_version:
return click.secho("No specified version found", fg=Color.yellow)
if version == -1:
versions = [specified_version]
else:
versions = await Aerich.filter(app=app, pk__gte=specified_version.pk)
for version in versions:
file = version.version
async with in_transaction(get_app_connection_name(config, app)) as conn:
file_path = os.path.join(Migrate.migrate_location, file)
content = get_version_content_from_file(file_path)
downgrade_query_list = content.get("downgrade")
if not downgrade_query_list:
return click.secho("No downgrade item found", fg=Color.yellow)
return click.secho("No downgrade items found", fg=Color.yellow)
for downgrade_query in downgrade_query_list:
await conn.execute_query(downgrade_query)
await last_version.delete()
return click.secho(f"Success downgrade {file}", fg=Color.green)
await version.delete()
if delete:
os.unlink(file_path)
click.secho(f"Success downgrade {file}", fg=Color.green)
@cli.command(help="Show current available heads in migrate location.")
@@ -149,7 +188,7 @@ async def heads(ctx: Context):
click.secho(version, fg=Color.green)
is_heads = True
if not is_heads:
click.secho("No available heads,try migrate", fg=Color.green)
click.secho("No available heads,try migrate first", fg=Color.green)
@cli.command(help="List all migrate items.")
@@ -171,12 +210,17 @@ async def history(ctx: Context):
help="Tortoise-ORM config module dict variable, like settings.TORTOISE_ORM.",
)
@click.option(
"--location", default="./migrations", show_default=True, help="Migrate store location."
"--location",
default="./migrations",
show_default=True,
help="Migrate store location.",
)
@click.pass_context
@coro
async def init(
ctx: Context, tortoise_orm, location,
ctx: Context,
tortoise_orm,
location,
):
config_file = ctx.obj["config_file"]
name = ctx.obj["name"]
@@ -217,9 +261,9 @@ async def init_db(ctx: Context, safe):
os.mkdir(dirname)
click.secho(f"Success create app migrate location {dirname}", fg=Color.green)
else:
return click.secho(f"Inited {app} already", fg=Color.yellow)
Migrate.write_old_models(config, app, location)
return click.secho(
f"Inited {app} already, or delete {dirname} and try again.", fg=Color.yellow
)
await Tortoise.init(config=config)
connection = get_app_connection(config, app)
@@ -228,15 +272,22 @@ async def init_db(ctx: Context, safe):
schema = get_schema_sql(connection, safe)
version = await Migrate.generate_version()
await Aerich.create(version=version, app=app)
with open(os.path.join(dirname, version), "w", encoding="utf-8") as f:
content = {
"upgrade": [schema],
}
json.dump(content, f, ensure_ascii=False, indent=2)
return click.secho(f'Success generate schema for app "{app}"', fg=Color.green)
await Aerich.create(
version=version,
app=app,
content=Migrate.get_models_content(config, app, location),
)
content = {
"upgrade": [schema],
}
write_version_file(os.path.join(dirname, version), content)
click.secho(f'Success generate schema for app "{app}"', fg=Color.green)
def main():
sys.path.insert(0, ".")
cli()
if __name__ == "__main__":
main()

View File

@@ -22,6 +22,9 @@ class BaseDDL:
_DROP_FK_TEMPLATE = 'ALTER TABLE "{table_name}" DROP FOREIGN KEY "{fk_name}"'
_M2M_TABLE_TEMPLATE = 'CREATE TABLE "{table_name}" ("{backward_key}" {backward_type} NOT NULL REFERENCES "{backward_table}" ("{backward_field}") ON DELETE CASCADE,"{forward_key}" {forward_type} NOT NULL REFERENCES "{forward_table}" ("{forward_field}") ON DELETE {on_delete}){extra}{comment};'
_MODIFY_COLUMN_TEMPLATE = 'ALTER TABLE "{table_name}" MODIFY COLUMN {column}'
_CHANGE_COLUMN_TEMPLATE = (
'ALTER TABLE "{table_name}" CHANGE {old_column_name} {new_column_name} {new_column_type}'
)
def __init__(self, client: "BaseDBAsyncClient"):
self.client = client
@@ -136,6 +139,16 @@ class BaseDDL:
new_column_name=new_column_name,
)
def change_column(
self, model: "Type[Model]", old_column_name: str, new_column_name: str, new_column_type: str
):
return self._CHANGE_COLUMN_TEMPLATE.format(
table_name=model._meta.db_table,
old_column_name=old_column_name,
new_column_name=new_column_name,
new_column_type=new_column_type,
)
def add_index(self, model: "Type[Model]", field_names: List[str], unique=False):
return self._ADD_INDEX_TEMPLATE.format(
unique="UNIQUE" if unique else "",

View File

@@ -13,7 +13,7 @@ class SqliteDDL(BaseDDL):
DIALECT = SqliteSchemaGenerator.DIALECT
def drop_column(self, model: "Type[Model]", column_name: str):
raise NotSupportError("Drop column is not support in SQLite.")
raise NotSupportError("Drop column is unsupported in SQLite.")
def modify_column(self, model: "Type[Model]", field_object: Field):
raise NotSupportError("Modify column is not support in SQLite.")
raise NotSupportError("Modify column is unsupported in SQLite.")

View File

@@ -1,24 +1,29 @@
import json
import inspect
import os
import re
from datetime import datetime
from importlib import import_module
from typing import Dict, List, Tuple, Type
from io import StringIO
from typing import Dict, List, Optional, Tuple, Type, Union
import click
from packaging import version
from packaging.version import LegacyVersion, Version
from tortoise import (
BackwardFKRelation,
BackwardOneToOneRelation,
BaseDBAsyncClient,
ForeignKeyFieldInstance,
ManyToManyFieldInstance,
Model,
Tortoise,
)
from tortoise.exceptions import OperationalError
from tortoise.fields import Field
from aerich.ddl import BaseDDL
from aerich.models import MAX_VERSION_LENGTH, Aerich
from aerich.utils import get_app_connection
from aerich.utils import get_app_connection, write_version_file
class Migrate:
@@ -39,31 +44,54 @@ class Migrate:
app: str
migrate_location: str
dialect: str
_db_version: Union[LegacyVersion, Version] = None
@classmethod
def get_old_model_file(cls):
return cls.old_models + ".py"
def get_old_model_file(cls, app: str, location: str):
return os.path.join(location, app, cls.old_models + ".py")
@classmethod
def get_all_version_files(cls) -> List[str]:
return sorted(
filter(lambda x: x.endswith("json"), os.listdir(cls.migrate_location)),
filter(lambda x: x.endswith("sql"), os.listdir(cls.migrate_location)),
key=lambda x: int(x.split("_")[0]),
)
@classmethod
async def get_last_version(cls) -> Aerich:
return await Aerich.filter(app=cls.app).first()
async def get_last_version(cls) -> Optional[Aerich]:
try:
return await Aerich.filter(app=cls.app).first()
except OperationalError:
pass
@classmethod
def remove_old_model_file(cls, app: str, location: str):
try:
os.unlink(cls.get_old_model_file(app, location))
except (OSError, FileNotFoundError):
pass
@classmethod
async def _get_db_version(cls, connection: BaseDBAsyncClient):
if cls.dialect == "mysql":
sql = "select version() as version"
ret = await connection.execute_query(sql)
cls._db_version = version.parse(ret[1][0].get("version"))
@classmethod
async def init_with_old_models(cls, config: dict, app: str, location: str):
migrate_config = cls._get_migrate_config(config, app, location)
await Tortoise.init(config=config)
last_version = await cls.get_last_version()
cls.app = app
cls.migrate_config = migrate_config
cls.migrate_location = os.path.join(location, app)
if last_version:
content = last_version.content
with open(cls.get_old_model_file(app, location), "w", encoding="utf-8") as f:
f.write(content)
await Tortoise.init(config=migrate_config)
migrate_config = cls._get_migrate_config(config, app, location)
cls.migrate_config = migrate_config
await Tortoise.init(config=migrate_config)
connection = get_app_connection(config, app)
cls.dialect = connection.schema_generator.DIALECT
@@ -79,6 +107,7 @@ class Migrate:
from aerich.ddl.postgres import PostgresDDL
cls.ddl = PostgresDDL(connection)
await cls._get_db_version(connection)
@classmethod
async def _get_last_version_num(cls):
@@ -93,8 +122,8 @@ class Migrate:
now = datetime.now().strftime("%Y%m%d%H%M%S").replace("/", "")
last_version_num = await cls._get_last_version_num()
if last_version_num is None:
return f"0_{now}_init.json"
version = f"{last_version_num + 1}_{now}_{name}.json"
return f"0_{now}_init.sql"
version = f"{last_version_num + 1}_{now}_{name}.sql"
if len(version) > MAX_VERSION_LENGTH:
raise ValueError(f"Version name exceeds maximum length ({MAX_VERSION_LENGTH})")
return version
@@ -102,12 +131,15 @@ class Migrate:
@classmethod
async def _generate_diff_sql(cls, name):
version = await cls.generate_version(name)
# delete if same version exists
for version_file in cls.get_all_version_files():
if version_file.startswith(version.split("_")[0]):
os.unlink(os.path.join(cls.migrate_location, version_file))
content = {
"upgrade": cls.upgrade_operators,
"downgrade": cls.downgrade_operators,
}
with open(os.path.join(cls.migrate_location, version), "w", encoding="utf-8") as f:
json.dump(content, f, indent=2, ensure_ascii=False)
write_version_file(os.path.join(cls.migrate_location, version), content)
return version
@classmethod
@@ -151,26 +183,6 @@ class Migrate:
else:
cls.downgrade_operators.append(operator)
@classmethod
def cp_models(
cls, app: str, model_files: List[str], old_model_file,
):
"""
cp currents models to old_model_files
:param app:
:param model_files:
:param old_model_file:
:return:
"""
pattern = rf"(\n)?('|\")({app})(.\w+)('|\")"
for i, model_file in enumerate(model_files):
with open(model_file, "r", encoding="utf-8") as f:
content = f.read()
ret = re.sub(pattern, rf"\2{cls.diff_app}\4\5", content)
mode = "w" if i == 0 else "a"
with open(old_model_file, mode, encoding="utf-8") as f:
f.write(f"{ret}\n")
@classmethod
def _get_migrate_config(cls, config: dict, app: str, location: str):
"""
@@ -189,7 +201,7 @@ class Migrate:
return config
@classmethod
def write_old_models(cls, config: dict, app: str, location: str):
def get_models_content(cls, config: dict, app: str, location: str):
"""
write new models to old models
:param config:
@@ -197,14 +209,26 @@ class Migrate:
:param location:
:return:
"""
cls.app = app
old_model_files = []
models = config.get("apps").get(app).get("models")
for model in models:
old_model_files.append(import_module(model).__file__)
cls.cp_models(app, old_model_files, os.path.join(location, app, cls.get_old_model_file()))
module = import_module(model)
possible_models = [getattr(module, attr_name) for attr_name in dir(module)]
for attr in filter(
lambda x: inspect.isclass(x) and issubclass(x, Model) and x is not Model,
possible_models,
):
file = inspect.getfile(attr)
if file not in old_model_files:
old_model_files.append(file)
pattern = rf"(\n)?('|\")({app})(.\w+)('|\")"
str_io = StringIO()
for i, model_file in enumerate(old_model_files):
with open(model_file, "r", encoding="utf-8") as f:
content = f.read()
ret = re.sub(pattern, rf"\2{cls.diff_app}\4\5", content)
str_io.write(f"{ret}\n")
return str_io.getvalue()
@classmethod
def diff_models(
@@ -288,13 +312,26 @@ class Migrate:
else:
is_rename = diff_key in cls._rename_new
if is_rename:
cls._add_operator(
cls._rename_field(new_model, old_field, new_field), upgrade,
)
if (
cls.dialect == "mysql"
and cls._db_version
and cls._db_version.major == 5
):
cls._add_operator(
cls._change_field(new_model, old_field, new_field),
upgrade,
)
else:
cls._add_operator(
cls._rename_field(new_model, old_field, new_field),
upgrade,
)
break
else:
cls._add_operator(
cls._add_field(new_model, new_field), upgrade, cls._is_fk_m2m(new_field),
cls._add_field(new_model, new_field),
upgrade,
cls._is_fk_m2m(new_field),
)
else:
old_field = old_fields_map.get(new_key)
@@ -310,7 +347,9 @@ class Migrate:
cls._add_operator(
cls._alter_null(new_model, new_field), upgrade=upgrade
)
if new_field.default != old_field.default:
if new_field.default != old_field.default and not callable(
new_field.default
):
cls._add_operator(
cls._alter_default(new_model, new_field), upgrade=upgrade
)
@@ -345,11 +384,15 @@ class Migrate:
if isinstance(new_field, ForeignKeyFieldInstance):
if old_field.db_constraint and not new_field.db_constraint:
cls._add_operator(
cls._drop_fk(new_model, new_field), upgrade, True,
cls._drop_fk(new_model, new_field),
upgrade,
True,
)
if new_field.db_constraint and not old_field.db_constraint:
cls._add_operator(
cls._add_fk(new_model, new_field), upgrade, True,
cls._add_fk(new_model, new_field),
upgrade,
True,
)
for old_key in old_keys:
@@ -359,12 +402,20 @@ class Migrate:
not upgrade and old_key not in cls._rename_new
):
cls._add_operator(
cls._remove_field(old_model, field), upgrade, cls._is_fk_m2m(field),
cls._remove_field(old_model, field),
upgrade,
cls._is_fk_m2m(field),
)
for new_index in new_indexes:
if new_index not in old_indexes:
cls._add_operator(cls._add_index(new_model, new_index,), upgrade)
cls._add_operator(
cls._add_index(
new_model,
new_index,
),
upgrade,
)
for old_index in old_indexes:
if old_index not in new_indexes:
cls._add_operator(cls._remove_index(old_model, old_index), upgrade)
@@ -460,6 +511,15 @@ class Migrate:
def _rename_field(cls, model: Type[Model], old_field: Field, new_field: Field):
return cls.ddl.rename_column(model, old_field.model_field_name, new_field.model_field_name)
@classmethod
def _change_field(cls, model: Type[Model], old_field: Field, new_field: Field):
return cls.ddl.change_column(
model,
old_field.model_field_name,
new_field.model_field_name,
new_field.get_for_dialect(cls.dialect, "SQL_TYPE"),
)
@classmethod
def _add_fk(cls, model: Type[Model], field: ForeignKeyFieldInstance):
"""

View File

@@ -6,6 +6,7 @@ MAX_VERSION_LENGTH = 255
class Aerich(Model):
version = fields.CharField(max_length=MAX_VERSION_LENGTH)
app = fields.CharField(max_length=20)
content = fields.TextField()
class Meta:
ordering = ["-id"]

View File

@@ -1,4 +1,5 @@
import importlib
from typing import Dict
from click import BadOptionUsage, Context
from tortoise import BaseDBAsyncClient, Tortoise
@@ -49,3 +50,46 @@ def get_tortoise_config(ctx: Context, tortoise_orm: str) -> dict:
ctx=ctx,
)
return config
_UPGRADE = "##### upgrade #####\n"
_DOWNGRADE = "##### downgrade #####\n"
def get_version_content_from_file(version_file: str) -> Dict:
"""
get version content
:param version_file:
:return:
"""
with open(version_file, "r", encoding="utf-8") as f:
content = f.read()
first = content.index(_UPGRADE)
second = content.index(_DOWNGRADE)
upgrade_content = content[first + len(_UPGRADE) : second].strip() # noqa:E203
downgrade_content = content[second + len(_DOWNGRADE) :].strip() # noqa:E203
ret = {"upgrade": upgrade_content.split("\n"), "downgrade": downgrade_content.split("\n")}
return ret
def write_version_file(version_file: str, content: Dict):
"""
write version file
:param version_file:
:param content:
:return:
"""
with open(version_file, "w", encoding="utf-8") as f:
f.write(_UPGRADE)
upgrade = content.get("upgrade")
if len(upgrade) > 1:
f.write(";\n".join(upgrade) + ";\n")
else:
f.write(f"{upgrade[0]};\n")
downgrade = content.get("downgrade")
if downgrade:
f.write(_DOWNGRADE)
if len(downgrade) > 1:
f.write(";\n".join(downgrade) + ";\n")
else:
f.write(f"{downgrade[0]};\n")

View File

@@ -13,10 +13,15 @@ from aerich.ddl.sqlite import SqliteDDL
from aerich.migrate import Migrate
db_url = os.getenv("TEST_DB", "sqlite://:memory:")
db_url_second = os.getenv("TEST_DB_SECOND", "sqlite://:memory:")
tortoise_orm = {
"connections": {"default": expand_db_url(db_url, True)},
"connections": {
"default": expand_db_url(db_url, True),
"second": expand_db_url(db_url_second, True),
},
"apps": {
"models": {"models": ["tests.models", "aerich.models"], "default_connection": "default"},
"models_second": {"models": ["tests.models_second"], "default_connection": "second"},
},
}
@@ -62,5 +67,5 @@ async def initialize_tests(event_loop, request):
Migrate.ddl = SqliteDDL(client)
elif client.schema_generator is AsyncpgSchemaGenerator:
Migrate.ddl = PostgresDDL(client)
Migrate.dialect = Migrate.ddl.DIALECT
request.addfinalizer(lambda: event_loop.run_until_complete(Tortoise._drop_databases()))

719
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,6 +1,6 @@
[tool.poetry]
name = "aerich"
version = "0.2.5"
version = "0.4.1"
description = "A database migrations tool for Tortoise ORM."
authors = ["long2ice <long2ice@gmail.com>"]
license = "Apache-2.0"
@@ -25,7 +25,7 @@ asyncpg = {version = "*", optional = true}
[tool.poetry.dev-dependencies]
flake8 = "*"
isort = "*"
black = "^19.10b0"
black = "^20.8b1"
pytest = "*"
pytest-xdist = "*"
pytest-asyncio = "*"

63
tests/models_second.py Normal file
View File

@@ -0,0 +1,63 @@
import datetime
from enum import IntEnum
from tortoise import Model, fields
class ProductType(IntEnum):
article = 1
page = 2
class PermissionAction(IntEnum):
create = 1
delete = 2
update = 3
read = 4
class Status(IntEnum):
on = 1
off = 0
class User(Model):
username = fields.CharField(max_length=20, unique=True)
password = fields.CharField(max_length=200)
last_login = fields.DatetimeField(description="Last Login", default=datetime.datetime.now)
is_active = fields.BooleanField(default=True, description="Is Active")
is_superuser = fields.BooleanField(default=False, description="Is SuperUser")
avatar = fields.CharField(max_length=200, default="")
intro = fields.TextField(default="")
class Email(Model):
email = fields.CharField(max_length=200)
is_primary = fields.BooleanField(default=False)
user = fields.ForeignKeyField("models_second.User", db_constraint=False)
class Category(Model):
slug = fields.CharField(max_length=200)
name = fields.CharField(max_length=200)
user = fields.ForeignKeyField("models_second.User", description="User")
created_at = fields.DatetimeField(auto_now_add=True)
class Product(Model):
categories = fields.ManyToManyField("models_second.Category")
name = fields.CharField(max_length=50)
view_num = fields.IntField(description="View Num")
sort = fields.IntField()
is_reviewed = fields.BooleanField(description="Is Reviewed")
type = fields.IntEnumField(ProductType, description="Product Type")
image = fields.CharField(max_length=200)
body = fields.TextField()
created_at = fields.DatetimeField(auto_now_add=True)
class Config(Model):
label = fields.CharField(max_length=200)
key = fields.CharField(max_length=20)
value = fields.JSONField()
status: Status = fields.IntEnumField(Status, default=Status.on)

View File

@@ -42,7 +42,7 @@ def test_create_table():
"id" SERIAL NOT NULL PRIMARY KEY,
"slug" VARCHAR(200) NOT NULL,
"name" VARCHAR(200) NOT NULL,
"created_at" TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
"user_id" INT NOT NULL REFERENCES "user" ("id") ON DELETE CASCADE
);
COMMENT ON COLUMN "category"."user_id" IS 'User';"""

View File

@@ -62,18 +62,18 @@ def test_sort_all_version_files(mocker):
mocker.patch(
"os.listdir",
return_value=[
"1_datetime_update.json",
"11_datetime_update.json",
"10_datetime_update.json",
"2_datetime_update.json",
"1_datetime_update.sql",
"11_datetime_update.sql",
"10_datetime_update.sql",
"2_datetime_update.sql",
],
)
Migrate.migrate_location = "."
assert Migrate.get_all_version_files() == [
"1_datetime_update.json",
"2_datetime_update.json",
"10_datetime_update.json",
"11_datetime_update.json",
"1_datetime_update.sql",
"2_datetime_update.sql",
"10_datetime_update.sql",
"11_datetime_update.sql",
]