KeyError Issue

[Creat-qc]

Describe the bug
2024-06-21 00:36:31,185 [ERROR] Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/tortoise/models.py", line 730, in _init_from_db
setattr(self, model_field, kwargs[key])
KeyError: 'tenant_id'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/sanic/resource_quota/components/quota/handler.py", line 1243, in get_tenant_admin_machine_info
tenant_machine_obj = await TenantMachineModel.filter(**query_kwargs).all()
File "/usr/local/lib/python3.9/site-packages/tortoise/queryset.py", line 1006, in _execute
instance_list = await self._db.executor_class(
File "/usr/local/lib/python3.9/site-packages/tortoise/backends/base/executor.py", line 138, in execute_select
instance: "Model" = self.model._init_from_db(
File "/usr/local/lib/python3.9/site-packages/tortoise/models.py", line 747, in _init_from_db
setattr(self, key, meta.fields_map[key].to_python_value(value))
KeyError: '1'

我们的多个项目都使用了tortoise-orm，但是仅有一个项目在一段时间运行后，就会大量的爆出这个错误，但是重启服务之后，这个问题就会被修复，我们比对了多个服务之间使用orm的差异，发现几乎一致，因此想问下这个错误出现的场景和原因应该是怎样的

[Creat-qc]

    except KeyError:
        self._partial = True
        # TODO: Apply similar perf optimisation as above for partial
        for key, value in kwargs.items():
            setattr(self, key, meta.fields_map[key].to_python_value(value))

和这里的TODO 有关系嘛，是不是一个已知的问题 @jairhenrique @kianmeng @isaquealves @wolph

[abondar]

These errors doesn't tell anything, apart from that you may be have different scheme at db and at model description

If you think there is bug - please provide reproducible self-sufficient example

[Creat-qc]

好的，我在你们的_init_from_db函数中加上了如下简单的代码注释

下图是日志的打印结果：

我们的models

我们的实际数据库表

可以看出实际数据库与models中的定义是一致的

且通过打印日志发现，正常情况下 map是正确的，但是在异常情况下，会引发不可恢复的致命问题

这是我截取的相关日志

下面是我对tortoise-orm 的相关实现的理解

通过修改源码，打印日志发现，应该是 需要特殊处理的参数 在做映射时出现了 找不到key 从而找不到 本应映射到的 参数类型，进而引发了 keyError问题；tortoise-orm 在处理model时，会做一层缓存机制，也就是将model中的key 应该映射成什么类型进行缓存。所以引发KeyError的本质原因有可能是 缓存中的map 与实际不一致，因此报错，并且在服务重启后，缓存清空，所以重启服务后可以正常运行一段时间。

为什么 会出现 COUNT(*)、数字 1 作为 Key 的情况

@jairhenrique @kianmeng @isaquealves @wolph @abondar

[abondar]

Well, in your logs, it shows that there is tenant_id present in kwargs, and it is present, so I don't know why error occurs, but at this point it doesn't looks like something tortoise responsible for

My guess would be that you can check unicodes of symbols in field name in code and in DB, may be there are symbols that looks identical, but in reality they are different unicode symbols

If it doesn't work - I still recommend to debug it around why it throws KeyError in kwargs, as it seems to be real issue here and everything else is just consequences of it

[Creat-qc]

好的 我会排查的 但排查的同时 我还是保留tortoise-orm 存在问题的观点，因为你还没有解释我上面的问题，keyerror为何会出现1 甚至于 出现了 表名称的字段，这看起来完全不像是 unicode 或编码问题。

并且，我理解 只要确定models中的表设计和数据库中实际的结构是一致的，那么理论上就不应该在这个init的函数中爆出错误了；也就是说只要我确定models是正确的，如果还是init函数中报错，就应该是tortoise-orm的问题了。如果达成这个共识后，我们排查的方向就更加明确了。

另外，想确定下 上一条评论中我的理解是正确的吗。

@abondar @grigi @zoliszeredi @AEnterprise @reedjosh

[abondar]

Well, it is indeed strange that it gives error about KeyError: 1, which I don't see how is happening with values that are seen in logged values
But there isn't really much I can do without reproducible example, that I would be able to run locally

So only option I see, as it is reproducible on your machine, is to run it with debugger and see concrete values and narrow it down to single values that are failing

If debugging is not option in this environment - you can add more logging, that will log all needed values moment before setattr operation is happening and see state of all variable at that exact moment

[Creat-qc]

好的 明天在工作的时间我会进行调试的。有任何新的进展我会同步在这里，如果你想到了任何 关于 Key 的异常情况（例如：Table、1、表字段），也请及时告知我，以便于及时修复线上问题

[waketzheng]

You should print the value of query_kwargs before line 1243:
File "/sanic/resource_quota/components/quota/handler.py", line 1243, in get_tenant_admin_machine_info
tenant_machine_obj = await TenantMachineModel.filter(**query_kwargs).all()

[Creat-qc]

@classmethod
    async def get_tenant_admin_machine_info(cls, current_user_info, args):
        tenant_id = current_user_info.get("tenantId")
        tenant_name = current_user_info.get("tenantName")
        query_kwargs = dict()
        query_kwargs["tenant_id"] = tenant_id

        for field_query_info_item in args.fieldQueryInfoList:
            query_kwargs[field_query_info_item.get("fieldName")] = field_query_info_item.get("fieldValue")

        try:
            tenant_machine_obj = await TenantMachineModel.filter(**query_kwargs).all()
        except Exception as e:
            raise custom_error(QuotaErrorCode.QUERY_MACHINE_USER_INFO.dict) from e

thanks to your reply, i will append it in our code. but in my opinion, its not the essential error. during this code, the param args was just checked by pydantic and passed to this func

[waketzheng]

Your code can be refactor to be:

    @classmethod
    async def get_tenant_admin_machine_info(cls, current_user_info, args):
        tenant_id = current_user_info.get("tenantId")
        tenant_name = current_user_info.get("tenantName")
        query_kwargs = dict(tenant_id=tenant_id, **{i.get('fieldName'): i.get('fieldValue') for i in args.fieldQueryInfoList})
        try:
            tenant_machine_obj = await TenantMachineModel.filter(**query_kwargs)
        except Exception as e:
            raise custom_error(QuotaErrorCode.QUERY_MACHINE_USER_INFO.dict) from e

[canghai118]

我们似乎也在线上出现了这个bug。 运行一段时间后报错KeyError。 报错的key和当前Model没关系，看起来来自于其它Model

@Creat-qc 你们后来是如何解决的？

[shenchucheng]

This is a bug in asyncmy. I have raised a PR attempting to fix this issue. You can check it out here: long2ice/asyncmy#108.

---

How to Reproduce the Issue in tortoise-orm:

1. Install MySQL server (mysqld).
2. Install asyncmy.
3. Run the following code:

import asyncio
from tortoise import Tortoise, fields, run_async
from tortoise.models import Model

class Event(Model):
    id = fields.IntField(True)

async def run():
    await Tortoise.init(db_url="mysql://root:123456@127.0.0.1:3306/test", modules={"models": ["__main__"]})
    await Tortoise.generate_schemas()

    await Event.create(id=1)
    await Event.create(id=2)

    async def query(i):
        event = await Event.filter(id=i).first()
        assert event.id == i, f"{event.id} != {i}"

    task = asyncio.create_task(query(1))
    await asyncio.sleep(0)
    task.cancel()
    
    try:
        await task
    except asyncio.CancelledError:
        pass
    
    # The assertion in the query function should pass, but it will raise an error with the message 'AssertionError: 1 != 2'.
    await query(2)

if __name__ == "__main__":
    run_async(run())