Nested async function scope not being maintained in use_user_data hook - not sure what changed

[shawncrawley]

I have been working on the refactor of reactpy-django to be compatible with the pre-lease v2 of reactpy core. In that context, I've been chasing down an issue related to the use_user_data hook. The pre-existing tests for that feature are failing. I wondered if it was due to underlying updates to reactpy core that I had missed in the refactor, but after digging a while, it seems to be possibly related to how more recent Python versions handle asynchronous variable scope.

Here's the component in question, from tests\test_app\components.py:

@component
def use_user_data():
    user_data_query, user_data_mutation = reactpy_django.hooks.use_user_data()
    user1 = reactpy_django.hooks.use_query(get_or_create_user1)
    user2 = reactpy_django.hooks.use_query(get_or_create_user2)
    current_user = reactpy_django.hooks.use_user()
    scope = reactpy_django.hooks.use_scope()

    async def login_user1(event):
        await login(scope, user1.data)
        user_data_query.refetch()
        user_data_mutation.reset()

    async def login_user2(event):
        await login(scope, user2.data)
        user_data_query.refetch()
        user_data_mutation.reset()

    async def logout_user(event):
        await logout(scope)
        user_data_query.refetch()
        user_data_mutation.reset()

    async def clear_data(event):
        user_data_mutation({})
        user_data_query.refetch()
        user_data_mutation.reset()

    async def on_submit(event):
        if event["key"] == "Enter":
            user_data_mutation((user_data_query.data or {}) | {event["target"]["value"]: event["target"]["value"]})

    return html.div(
        {
            "id": "use-user-data",
            "data-success": bool(user_data_query.data),
            "data-fetch-error": bool(user_data_query.error),
            "data-mutation-error": bool(user_data_mutation.error),
            "data-loading": user_data_query.loading or user_data_mutation.loading,
            "data-username": ("AnonymousUser" if current_user.is_anonymous else current_user.username),
        },
        html.div("use_user_data"),
        html.button({"className": "login-1", "onClick": login_user1}, "Login 1"),
        html.button({"className": "login-2", "onClick": login_user2}, "Login 2"),
        html.button({"className": "logout", "onClick": logout_user}, "Logout"),
        html.button({"className": "clear", "onClick": clear_data}, "Clear Data"),
        html.div(f"User: {current_user}"),
        html.div(f"Data: {user_data_query.data}"),
        html.div(f"Data State: (loading={user_data_query.loading}, error={user_data_query.error})"),
        html.div(f"Mutation State: (loading={user_data_mutation.loading}, error={user_data_mutation.error})"),
        html.div(html.input({"onKeyPress": on_submit, "placeholder": "Type here to add data"})),
    )

Here's the test code from tests\test_app\tests\test_components.py:

@navigate_to_page("/")
    def test_component_use_user_data(self):
        text_input = self.page.wait_for_selector("#use-user-data input")
        login_1 = self.page.wait_for_selector("#use-user-data .login-1")
        login_2 = self.page.wait_for_selector("#use-user-data .login-2")
        logout = self.page.wait_for_selector("#use-user-data .logout")
        clear = self.page.wait_for_selector("#use-user-data .clear")

        # Test AnonymousUser data
        user_data_div = self.page.wait_for_selector(
            "#use-user-data[data-success=false][data-fetch-error=false][data-mutation-error=false][data-loading=false][data-username=AnonymousUser]"
        )
        assert "Data: None" in user_data_div.text_content()

        # Test first user's data
        login_1.click(delay=CLICK_DELAY)
        user_data_div = self.page.wait_for_selector(
            "#use-user-data[data-success=false][data-fetch-error=false][data-mutation-error=false][data-loading=false][data-username=user_1]"
        )
        assert "Data: {}" in user_data_div.text_content()
        text_input.type("test", delay=CLICK_DELAY)
        text_input.press("Enter", delay=CLICK_DELAY)
        user_data_div = self.page.wait_for_selector(
            "#use-user-data[data-success=true][data-fetch-error=false][data-mutation-error=false][data-loading=false][data-username=user_1]"
        )
        assert "Data: {'test': 'test'}" in user_data_div.text_content()

Here's the code for the hook where the bug seems to exist (at src\reactpy_django\hooks.py):

def use_user_data(
    default_data: (None | dict[str, Callable[[], Any] | Callable[[], Awaitable[Any]] | Any]) = None,
    save_default_data: bool = False,
) -> UserData:
    """Get or set user data stored within the REACTPY_DATABASE.

    Kwargs:
        default_data: A dictionary containing `{key: default_value}` pairs. \
            For computationally intensive defaults, your `default_value` \
            can be sync or async functions that return the value to set.
        save_default_data: If True, `default_data` values will automatically be stored \
            within the database if they do not exist.
    """
    from reactpy_django.models import UserDataModel

    user = use_user()

    async def _set_user_data(data: dict):
        if not isinstance(data, dict):
            msg = f"Expected dict while setting user data, got {type(data)}"
            raise TypeError(msg)
        if user.is_anonymous:
            msg = "AnonymousUser cannot have user data."
            raise ValueError(msg)

        pk = get_pk(user)
        model, _ = await UserDataModel.objects.aget_or_create(user_pk=pk)
        model.data = orjson.dumps(data)
        await model.asave()

    query: Query[dict | None] = use_query(
        _get_user_data,
        kwargs={
            "user": user,
            "default_data": default_data,
            "save_default_data": save_default_data,
        },
        postprocessor=None,
    )
    mutation = use_mutation(_set_user_data, refetch=_get_user_data)

    return UserData(query, mutation)

The test, which can also be reproduced manually, loads the component initially with no user logged in. Then, a user is logged in by clicking the "Login 1" button. Then, "test" is entered into the text input, and "Enter" is hit to submit. This should then apply/save "test" to the logged-in user's data. Instead, the ValueError("AnonymousUser cannot have user data.") is thrown.

So it seems that the user variable from its defined scope (the use_user_data function) is not being properly shared/persisted within the scope of the nested async _set_user_data function. I verified that the user variable updates and is reflected properly in the parent scope after logging in user_1, but again, the nested function's scope will still reflect an anonymous user.

I was able to fix (?) this issue - or at least get around it - by updating the use_user_data to also store the user as as a reference variable with use_ref, like so:

def use_user_data(
    default_data: (None | dict[str, Callable[[], Any] | Callable[[], Awaitable[Any]] | Any]) = None,
    save_default_data: bool = False,
) -> UserData:
    """Get or set user data stored within the REACTPY_DATABASE.

    Kwargs:
        default_data: A dictionary containing `{key: default_value}` pairs. \
            For computationally intensive defaults, your `default_value` \
            can be sync or async functions that return the value to set.
        save_default_data: If True, `default_data` values will automatically be stored \
            within the database if they do not exist.
    """
    from reactpy_django.models import UserDataModel

    user = use_user()
    user_ref = use_ref(user)
    user_ref.current = user  # Always update the ref to the latest user

    async def _set_user_data(data: dict):
        current_user = user_ref.current
        if not isinstance(data, dict):
            msg = f"Expected dict while setting user data, got {type(data)}"
            raise TypeError(msg)
        if current_user.is_anonymous:
            msg = "AnonymousUser cannot have user data."
            raise ValueError(msg)

        pk = get_pk(current_user)
        model, _ = await UserDataModel.objects.aget_or_create(user_pk=pk)
        model.data = orjson.dumps(data)
        await model.asave()

I'm wondering if there's a better way, and if someone else knows better what's going on here. I'm guessing there was an update in the way that a recent version of Python handles scope in this type of context. I did experience something very similar recently where some similar code had worked before and then stopped working "randomly" (or so I thought, but I'm pretty sure I did update my Python version in between).

[shawncrawley]

As a follow-up, I thought about it more and definitely experienced a similar issue recently. I had a component with a nested function that I actually called in a background thread (using python's threading module). In that case, I was just trying to access and update a state variable (from use_state) in the background thread. It worked when I initially wrote the code, but months later (and I believe a Python update later), it started failing with the same thing we're seeing here - the state variable wasn't being updated/referenced/synchronized appropriately. I ended up fixing that issue by getting a reference to the current state from the functional setter (i.e. set_my_variable(lambda current: current + 1)) rather than by referencing the referenced variable itself (i.e. set_my_variable(my_variable + 1)).

Hopefully this can help in getting to the bottom of all this.

[Archmonger]

You can verify if this was caused by a specific Python version by forcing the test suite to use a specific Python version:

# Install/update Python 3.10 on your machine, if needed
hatch python install 3.10

# Run the test suite with Python 3.10. Only execute tests that contain the keyword "user_data" in the name.
hatch test --python 3.10 -k "user_data"

I would not be entirely surprised if the structural changes I've made to use_effect and use_async_effect ultimately caused this. First, try running on Python 3.10 or below and let me know if it fixes things for you.