[BUG] Possible repeated frame/episode counting and epsilon annealing in getting-started-5 tutorial

[17718177956]

Description

In tutorials/sphinx-tutorials/getting-started-5.py, the following
operations are placed inside the optimization loop:

for _ in range(optim_steps):
    sample = rb.sample(128)
    loss_vals = loss(sample)
    loss_vals["loss"].backward()
    optim.step()
    optim.zero_grad()

    exploration_module.step(data.numel())
    updater.step()

    total_count += data.numel()
    total_episodes += data["next", "done"].sum()

My understanding is that data is collected once per outer collector
iteration, while the inner loop only performs multiple optimization steps
using replay-buffer samples.
With:
frames_per_batch = 100
optim_steps = 10
one collector iteration produces approximately 100 new frames, but:
total_count += data.numel()
is executed 10 times, so total_count increases by approximately 1000.

Similarly:

completed episodes in the same data batch are counted 10 times;
exploration_module.step(data.numel()) advances epsilon annealing 10
times for the same collected frames;
frames and episodes collected during the initial replay-buffer warm-up
are not counted because these statements are inside
if len(rb) > init_rand_steps.

There may also be an unrelated logging-condition issue:
if i % 10:
This executes for most iterations rather than every tenth iteration.
Was the intended condition:
if i % 10 == 0:
Expected behavior

Environment-related accounting and epsilon annealing should seemingly be
performed once per collected batch:
for i, data in enumerate(collector):
num_frames = data.numel()
rb.extend(data)

total_count += num_frames
total_episodes += data["next", "done"].sum().item()
exploration_module.step(num_frames)

if len(rb) > init_rand_steps:
    for _ in range(optim_steps):
        sample = rb.sample(128)
        loss_vals = loss(sample)

        optim.zero_grad()
        loss_vals["loss"].backward()
        optim.step()
        updater.step()

updater.step() may reasonably remain inside the optimization loop,
because it follows each online-network update.

Question

Is the current placement intentional, or should the environment-frame
accounting, episode accounting, and epsilon annealing be moved outside the
optimization loop?