Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions cli/cmd/mcp_tools_runs_activity.go
Original file line number Diff line number Diff line change
Expand Up @@ -465,8 +465,8 @@ var mcpCaveatsRunsActivity = []mcpCaveat{
mcpAnchoredRef(mcpJava+"security/GrpcApiKeyInterceptor.java:67-70,100,156-159",
"static final Map<String, String> AUDITED", "audit(action, method, Status.Code.PERMISSION_DENIED",
"private void audit("),
mcpAnchoredRef(mcpJava+"schedule/TestScheduler.java:118-124", "Actor.SCHEDULER"),
mcpAnchoredRef(mcpJava+"disruption/DisruptionScheduler.java:124-130", "Actor.SCHEDULER"),
mcpAnchoredRef(mcpJava+"schedule/TestScheduler.java:117-123", "Actor.SCHEDULER"),
mcpAnchoredRef(mcpJava+"disruption/DisruptionScheduler.java:129-135", "Actor.SCHEDULER"),
mcpAnchoredRef(mcpJava+"disruption/DisruptionResource.java:93", "auditTrail.done();"),
"kates/src/main/resources/db/migration/V26__audit_actor.sql:1-8",
},
Expand All @@ -488,7 +488,7 @@ var mcpCaveatsRunsActivity = []mcpCaveat{
mcpJava + "disruption/DisruptionReportRepository.java:20-43",
mcpJava + "disruption/DisruptionOrphanReconciler.java:334-368",
mcpJava + "disruption/DisruptionTemplateResource.java:70-72",
mcpJava + "disruption/DisruptionScheduler.java:112-124",
mcpJava + "disruption/DisruptionScheduler.java:117-129",
mcpJava + "disruption/DisruptionReportEntity.java:39-47",
mcpAnchoredRef(mcpJava+"disruption/DisruptionAnalysisResource.java:118-203",
`@Path("/compound")`, `"results", outcome.results()`),
Expand Down
4 changes: 3 additions & 1 deletion docs/book/11-api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ curl -s -X PUT -H "X-API-Key: $KATES_API_KEY" -H "Content-Type: application/json
While the API is draining:

- Every request that would start a test run, a disruption, a compound run or a resilience test is answered `503 Service Unavailable` with a `Retry-After: 60` header. Two answers differ: a full engine still answers `429`, and `POST /api/tests/bulk` answers `202` with every item failed.
- Test and disruption schedules don't fire. A firing that falls inside the drain is skipped, not run later.
- Test and disruption schedules don't fire. A due time inside the drain fires when the drain ends, if that is within 5 minutes of it, and is skipped otherwise.
- Live runs and faults go on to their end. A resilience test admitted before the drain still starts its benchmark when it reaches it. A dry run, a cancel and every read are answered as usual.

The switch is kept in memory: a restarted pod is not draining. A body without `draining` is a `400`.
Expand Down Expand Up @@ -1379,6 +1379,8 @@ It takes these query parameters:

A schedule stores a test request with a cron expression, and while the schedule is enabled the Kates API submits that request as a new run each time the expression fires. `kates schedule` calls every one of these endpoints except `PUT`.

Each due time of a test or disruption schedule fires once, on the member that leads, within 10 seconds of it, or up to 5 minutes late after a restart, a takeover or a drain. When several were missed, only the latest starts a run. A `PUT` that turns a schedule on or changes its cron passes over the due times before it. A test run a schedule starts carries the Idempotency-Key `schedule:<id>:<due time>`, such as `schedule:b5a9a9bb:2026-11-15T10:37:00Z`, so `GET /api/tests?clientKey=` finds it by that key.

#### POST /api/schedules

Create a recurring test schedule. Cron expressions use the standard 5-field Unix format (`minute hour day-of-month month day-of-week`, evaluated in UTC), at most 64 characters: `*`, numbers, ranges (`9-17`), lists (`0,30`) and steps (`*/15`, `0-30/5`), with month and day names such as `JAN` or `MON-FRI`. Day-of-week `0` and `7` are both Sunday, and when both day fields are set the schedule fires on a day that matches either one. Macros such as `@daily`, six-field expressions with seconds, and Quartz's `?` and `L` are refused with a `400` keyed `cronExpression` in `fieldErrors`, and the schedule isn't saved.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
import java.time.Instant;
import jakarta.persistence.*;

import com.fasterxml.jackson.annotation.JsonIgnore;

/**
* JPA entity for a scheduled/recurring disruption test.
* Supports either a playbook name or a raw plan JSON.
Expand Down Expand Up @@ -36,6 +38,11 @@ public class DisruptionScheduleEntity {
@Column(name = "last_run_at")
private Instant lastRunAt;

/** Written only by ScheduleFirings, in SQL, so a save of a stale copy never takes a claim back. */
@JsonIgnore
@Column(name = "last_due", insertable = false, updatable = false)
private Instant lastDue;

@Column(name = "created_at", nullable = false)
private Instant createdAt;

Expand Down Expand Up @@ -114,4 +121,8 @@ public Instant getCreatedAt() {
public void setCreatedAt(Instant createdAt) {
this.createdAt = createdAt;
}

public Instant getLastDue() {
return lastDue;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ public class DisruptionScheduleResource {
@Inject
ObjectMapper objectMapper;

@Inject
com.bmscomp.kates.schedule.ScheduleFirings firings;

@GET
@Operation(summary = "List disruption schedules")
public Response listSchedules(
Expand Down Expand Up @@ -154,6 +157,8 @@ public Response updateSchedule(
return badCron.get();
}
}
boolean passOver = (request.enabled == Boolean.TRUE && !entity.isEnabled())
|| (request.cronExpression != null && !request.cronExpression.equals(entity.getCronExpression()));
if (request.name != null) entity.setName(request.name);
if (request.cronExpression != null) entity.setCronExpression(request.cronExpression);
if (request.enabled != null) entity.setEnabled(request.enabled);
Expand All @@ -168,6 +173,10 @@ public Response updateSchedule(
}
}
em.merge(entity);
if (passOver) {
// Turning a schedule on or re-timing it passes over the due times before now.
firings.claim(com.bmscomp.kates.schedule.ScheduleFirings.Kind.DISRUPTIONS, id, java.time.Instant.now());
}
return Response.ok(entity).build();
}

Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
package com.bmscomp.kates.disruption;

import java.time.Instant;
import java.time.ZoneOffset;
import java.time.ZonedDateTime;
import java.util.List;
import java.util.UUID;
import jakarta.enterprise.context.ApplicationScoped;
Expand All @@ -14,12 +12,10 @@
import io.quarkus.scheduler.Scheduled;
import org.jboss.logging.Logger;

import com.bmscomp.kates.schedule.TestScheduler;

/**
* Cron-based scheduler for recurring disruption tests.
* Evaluates enabled schedules every 60 seconds, executing matching plans
* via DisruptionOrchestrator or from the playbook catalog.
* Evaluates enabled schedules every 10 seconds, and runs the plan of each due
* time once, via DisruptionOrchestrator or from the playbook catalog.
*/
@ApplicationScoped
public class DisruptionScheduler {
Expand Down Expand Up @@ -48,39 +44,48 @@ public class DisruptionScheduler {
ObjectMapper objectMapper;

@jakarta.inject.Inject
com.bmscomp.kates.service.SchedulerLeaseService leases;
com.bmscomp.kates.schedule.ScheduleFirings firings;

@jakarta.inject.Inject
com.bmscomp.kates.quorum.Quorum quorum;

@Scheduled(every = "60s", identity = "kates-disruption-schedule-evaluator")
// SKIP: a tick runs each plan it fires to its end. A tick started meanwhile would fire a
// schedule due the same minute into the target the first plan holds, and spend its due time
// on a REJECTED report; with SKIP, the running tick fires it after its plan, as one tick did.
@Scheduled(
every = "10s",
identity = "kates-disruption-schedule-evaluator",
concurrentExecution = Scheduled.ConcurrentExecution.SKIP)
void evaluateSchedules() {
// A draining instance fires nothing, and leaves the lease to one that may; so does a member
// that does not lead its quorum.
evaluate(Instant.now());
}

/**
* One tick at {@code now}; package-private, so a test ticks at the time it
* needs. Each due time runs once, claimed as TestScheduler claims a test's.
*/
void evaluate(Instant now) {
// A draining instance fires nothing; nor does a member that does not lead its quorum.
if (concurrencyGuard.isDraining() || !quorum.acceptsWrites()) {
return;
}
// Only the lease holder may trigger disruptions — with replicas > 1
// each schedule would otherwise inject faults once per replica.
if (!leases.tryAcquire("disruption-scheduler", java.time.Duration.ofSeconds(55))) {
return;
}
List<DisruptionScheduleEntity> schedules = em.createQuery(
"SELECT s FROM DisruptionScheduleEntity s WHERE s.enabled = true",
DisruptionScheduleEntity.class)
.getResultList();

if (schedules.isEmpty()) return;

ZonedDateTime now = ZonedDateTime.now(ZoneOffset.UTC);
LOG.debugf("Evaluating %d disruption schedules at %s", schedules.size(), now);

for (DisruptionScheduleEntity schedule : schedules) {
try {
if (TestScheduler.matchesCron(schedule.getCronExpression(), now)) {
LOG.info("Disruption schedule '" + schedule.getName() + "' triggered at " + now);
executeSchedule(schedule);
}
Instant since = com.bmscomp.kates.schedule.DueTimes.since(
schedule.getCreatedAt(), schedule.getLastDue(), schedule.getLastRunAt());
com.bmscomp.kates.schedule.DueTimes.toFire(schedule.getCronExpression(), since, now)
.filter(due -> firings.claim(
com.bmscomp.kates.schedule.ScheduleFirings.Kind.DISRUPTIONS, schedule.getId(), due))
.ifPresent(due -> {
LOG.info("Disruption schedule '" + schedule.getName() + "' triggered for " + due);
executeSchedule(schedule);
});
} catch (Exception e) {
LOG.warn("Failed to evaluate disruption schedule '" + schedule.getName() + "'", e);
}
Expand Down
53 changes: 53 additions & 0 deletions kates/src/main/java/com/bmscomp/kates/schedule/DueTimes.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
package com.bmscomp.kates.schedule;

import java.time.Duration;
import java.time.Instant;
import java.time.ZoneOffset;
import java.time.ZonedDateTime;
import java.time.temporal.ChronoUnit;
import java.util.Optional;

import com.cronutils.model.time.ExecutionTime;

/**
* Which due time of a schedule fires now. A tick that came late, after a
* takeover or a restart, still fires the due time it missed, up to
* {@link #CATCH_UP} late; when it missed several, only the latest starts a run.
*/
public final class DueTimes {

/** How late a due time may still fire. */
static final Duration CATCH_UP = Duration.ofMinutes(5);

private DueTimes() {}

/**
* The instant a schedule's due times are counted after: the latest of its
* creation, its last claimed due time and its last run's start. The last
* run counts too, since a row from before V31 has no last_due, and a
* Kates API of the version before it still fires during a rolling upgrade.
*/
public static Instant since(Instant createdAt, Instant lastDue, Instant lastRunAt) {
Instant since = createdAt;
for (Instant later : new Instant[] {lastDue, lastRunAt}) {
if (later != null && later.isAfter(since)) {
since = later;
}
}
return since;
}

/**
* The latest due time {@code d} of {@code cron} (UNIX, UTC) with
* {@code since < d <= now} and {@code d >= now - CATCH_UP}, or empty.
* Throws IllegalArgumentException when {@code cron} does not parse.
*/
public static Optional<Instant> toFire(String cron, Instant since, Instant now) {
ExecutionTime times = ExecutionTime.forCron(TestScheduler.CRON.parse(cron));
ZonedDateTime minute = now.atZone(ZoneOffset.UTC).truncatedTo(ChronoUnit.MINUTES);
// lastExecution is strictly before its argument, so a due time at this very minute is asked first.
Optional<ZonedDateTime> latest = times.isMatch(minute) ? Optional.of(minute) : times.lastExecution(minute);
return latest.map(ZonedDateTime::toInstant)
.filter(due -> due.isAfter(since) && !due.isBefore(now.minus(CATCH_UP)));
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
package com.bmscomp.kates.schedule;

import java.time.Instant;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import jakarta.persistence.EntityManager;
import jakarta.transaction.Transactional;

/**
* Claims a schedule's due time in the database. Two members that both believe
* they lead, a paused leader and the one that took over, may both tick; the
* conditional update lets only one of them move last_due to the due time, and
* only that one fires it.
*/
@ApplicationScoped
public class ScheduleFirings {

/** The tables that hold schedules; the only table names the claim's SQL takes. */
public enum Kind {
TESTS("scheduled_test_runs"),
DISRUPTIONS("disruption_schedules");

private final String table;

Kind(String table) {
this.table = table;
}
}

@Inject
EntityManager em;

/**
* Moves the schedule's last_due forward to {@code due}; true when this call
* moved it, and so fires that due time. A database error propagates: the
* due time then fires nowhere, rather than perhaps twice.
*/
@Transactional(Transactional.TxType.REQUIRES_NEW)
public boolean claim(Kind kind, String id, Instant due) {
return em.createNativeQuery("UPDATE " + kind.table + " SET last_due = :due"
+ " WHERE id = :id AND (last_due IS NULL OR last_due < :due)")
.setParameter("due", due)
.setParameter("id", id)
.executeUpdate()
> 0;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ public class ScheduleResource {
@Inject
Validator validator;

@Inject
ScheduleFirings firings;

/**
* A 400 naming each field of the schedule's test request that the Kates
* API would refuse, by its path under {@code testRequest}, or empty when
Expand Down Expand Up @@ -242,6 +245,9 @@ public Response updateSchedule(
return refused.get();
}
}
boolean passOver = (request.enabled == Boolean.TRUE && !schedule.isEnabled())
|| (request.cronExpression != null
&& !request.cronExpression.equals(schedule.getCronExpression()));
if (request.name != null) schedule.setName(request.name);
if (request.cronExpression != null) schedule.setCronExpression(request.cronExpression);
if (request.enabled != null) schedule.setEnabled(request.enabled);
Expand All @@ -252,6 +258,11 @@ public Response updateSchedule(
throw new RuntimeException(e);
}
}
if (passOver) {
// Turning a schedule on or re-timing it passes over the due times before now,
// before the save, so a tick between the two still sees it as it was.
firings.claim(ScheduleFirings.Kind.TESTS, id, java.time.Instant.now());
}
repository.save(schedule);
return Response.ok(schedule).build();
})
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
import jakarta.persistence.Id;
import jakarta.persistence.Table;

import com.fasterxml.jackson.annotation.JsonIgnore;

/**
* JPA entity for a scheduled/recurring test configuration.
* Stores the cron expression and the serialized CreateTestRequest.
Expand Down Expand Up @@ -36,6 +38,11 @@ public class ScheduledTestRun {
@Column(name = "last_run_at")
private Instant lastRunAt;

/** Written only by ScheduleFirings, in SQL, so a save of a stale copy never takes a claim back. */
@JsonIgnore
@Column(name = "last_due", insertable = false, updatable = false)
private Instant lastDue;

@Column(name = "created_at", nullable = false)
private Instant createdAt;

Expand Down Expand Up @@ -106,4 +113,8 @@ public Instant getCreatedAt() {
public void setCreatedAt(Instant createdAt) {
this.createdAt = createdAt;
}

public Instant getLastDue() {
return lastDue;
}
}
Loading
Loading