The reward implementation is inconsistent with the description in the paper.

[JMZhou123]

Hi! Thanks for your great work, but there are some problem about the GRPO reward. While the paper describes a process reward using scores across 8 dimensions, the actual code provides doctor_reward (5 dimensions) and doctor_reward_v2 (7 dimensions). Furthermore, the reward weights do not align with the paper, and the code defaults to calling doctor_reward. Additionally, only the final process reward of whole trajectory(not turn by turn) is used for loss calculation, which seems to deviate from the core logic proposed in the paper.

[thu-unicorn]

Thank you so much for taking the time to review our codebase.

1. Reward Dimensions and Weights
We apologize for the confusion here. The both reward functions that were initially uploaded were older versions used during our ablation studies. We have just updated the repository to include the correct reward implementation in verl/utils/reward_score/doctor_reward.py. This updated version strictly follows the 8 dimensions, and the hierarchical veto system exactly as described in the paper.

2. Turn-by-Turn vs. Final Trajectory Reward
Regarding your second point, there might be a slight misunderstanding of how the rollout collects rewards. The process reward is actually calculated turn-by-turn, not just once at the end of the whole trajectory.

Our implementation is built on top of the official veRL framework (https://github.com/verl-project/verl). During the multi-turn interaction loop, the Consultation Evaluator evaluates the assistant's and output immediately after every single turn, and appends it to a reward list.

Here is how it works under the hood in our sglang rollout implementation:

In doctor_interaction.py, every time the doctor agent generates a new response, we immediately trigger calculate_score() for that specific turn:

async def generate_response(self, instance_id: str, messages: list[dict[str, Any]], **kwargs):
    # ... extract the latest assistant reply ...
    
    # Calculate the reward for this specific turn
    if answer_content or think_content:
        reward = await self.calculate_score(instance_id)
        # ...
        return should_terminate_sequence, patient_reply, reward, {}

Then, in the main interaction loop inside sglang_rollout.py (following veRL's asynchronous rollout design), this turn-level reward is continuously appended to user_turn_rewards:

# Inside the Interacting state loop in sglang_rollout.py
interaction = self.interaction_map[interaction_name]

# 1. Generate patient response and get the doctor's reward for the current turn
should_terminate_sequence, content, reward, metrics = await interaction.generate_response(
    _req.request_id, messages, **_req.interaction_kwargs
)

# 2. Append the current turn's reward to the list
user_turn_rewards.append(reward)

if should_terminate_sequence:
    finish_reason_type = FinishReasonTypeEnum.STOP
    _req.state = AsyncRolloutRequestStateEnum.COMPLETED
    break

Finally, when the episode terminates, the entire array of user_turn_rewards = [r1, r2, r3, ...] is saved. This provides the dense, step-by-step feedback signals required for the GRPO algorithm, exactly as conceptualized in our paper.