Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions domains/cognition/towers/examples/margin.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
//! Which margin the commander should ask of the lookahead's proposals, on seeds chosen in advance:
//! `cargo run --release -p cognition_towers --example margin [SEEDS] [OFFSET] [MARGIN...]`
//! plays seeds `OFFSET+1 ..= OFFSET+SEEDS` (default 400 from 0). Each margin is played against a
//! commander with no advisor, and against a commander with the lookahead at margin 2 (the head to
//! head), paired seeds, both sides. The margin of 5 was found on seeds 1..=400, so a test of it is
//! made on others (`margin 800 10000`).

use cognition_towers::{Captain, Comparison, compare_captains};
use recreation_towers::{Commander, Lookahead, Rules};

fn main() -> Result<(), String> {
let mut args = std::env::args().skip(1);
let n: u64 = args.next().and_then(|s| s.parse().ok()).unwrap_or(400);
let offset: u64 = args.next().and_then(|s| s.parse().ok()).unwrap_or(0);
let mut margins: Vec<f32> = args.filter_map(|s| s.parse().ok()).collect();
if margins.is_empty() {
margins = vec![2.0, 3.0, 5.0, 7.0, 10.0];
}
let rules = Rules::default();
let seeds: Vec<u64> = (offset + 1..=offset + n).collect();
let with = |m: f32| -> Box<dyn Captain> {
Box::new(
Commander::new(rules).with_lookahead(false).with_advisor(Lookahead::new(rules), 0.0).with_advice_margin(m),
)
};
let alone = || -> Box<dyn Captain> { Box::new(Commander::new(rules).with_lookahead(false)) };
println!("margin: seeds {}..={} ({} games each)\n", offset + 1, offset + n, 2 * n);
println!(
"| margin | against no advisor | won | lost | against margin 2 | won | lost |\n|---|---|---|---|---|---|---|"
);
for m in margins {
let a = compare_captains(rules, &|| with(m), &alone, &seeds)?;
let b = compare_captains(rules, &|| with(m), &|| with(2.0), &seeds)?;
let Comparison { margin, error, won, lost, .. } = a;
println!(
"| {m} | {margin:+.3} ± {error:.3} | {won} | {lost} | {:+.3} ± {:.3} (lower {:+.3}) | {} | {} |",
b.margin,
b.error,
b.margin - 1.96 * b.error,
b.won,
b.lost
);
}
Ok(())
}
2 changes: 1 addition & 1 deletion domains/recreation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ commander's loop: each runs on a thread of its own at the station behind a mailb
(`recreation_towers_ground::advisors::Runner`: the newest map and hand go in, older ones that were
not looked at are dropped, proposals come out when they are ready). The computer commander puts
what has arrived in hand with `Commander::advise` before each decision, takes the best proposal
that gains more than 2 points, is not stale and still fits, and then shows the advisors the map
that gains more than 5 points, is not stale and still fits, and then shows the advisors the map
and its hand. It never waits. Which advisor each computer team has:

```bash
Expand Down
15 changes: 8 additions & 7 deletions domains/recreation/towers/src/commander.rs
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,8 @@ const CRUISE: f32 = 16.0;
const REFUEL_MARGIN: f32 = 15.0;
const BACK_TO_WORK: f32 = 0.9;
/// A commander takes a proposal of an advisor only if it gains more than this many points.
pub const ADVICE_MARGIN: f32 = 2.0;
/// 5, not 2: on 800 fresh seeds it beat 2 head to head (plans/actors.md, step 4).
pub const ADVICE_MARGIN: f32 = 5.0;

/// A computer in a team's seat.
#[derive(Clone, Debug)]
Expand Down Expand Up @@ -743,7 +744,7 @@ mod tests {
let g = Game::new(Rules::default());
let red_home = Order::Take(g.tower("Red home"));
let mut c = bare(&g);
c.advise([proposal(0, red_home, 5.0)]);
c.advise([proposal(0, red_home, 8.0)]);
c.decide(&g.view(Team::Blue));
assert_eq!(c.order(0), Some(red_home));
}
Expand Down Expand Up @@ -806,7 +807,7 @@ mod tests {
let g = Game::new(Rules::default());
let (a, b) = (Order::Take(g.tower("Red home")), Order::Take(g.tower("east")));
let mut c = bare(&g);
c.advise([proposal(0, a, 4.0), proposal(0, b, 6.0), proposal(2, Order::Hold, 3.0)]);
c.advise([proposal(0, a, 7.0), proposal(0, b, 9.0), proposal(2, Order::Hold, 6.0)]);
c.decide(&g.view(Team::Blue));
assert_eq!((c.order(0), c.order(2) == Some(Order::Hold)), (Some(b), false));
c.decide(&g.view(Team::Blue));
Expand All @@ -819,7 +820,7 @@ mod tests {
let red_home = Order::Take(g.tower("Red home"));
let mut c = bare(&g);
let q = vec![Queued::done(Order::Home)];
c.advise([Proposal { queue: q.clone(), ..proposal(0, red_home, 5.0) }]);
c.advise([Proposal { queue: q.clone(), ..proposal(0, red_home, 8.0) }]);
c.decide(&g.view(Team::Blue));
assert_eq!(c.advised_queue(0), q);
assert!(c.advised_queue(1).is_empty());
Expand All @@ -829,7 +830,7 @@ mod tests {
fn an_advisor_that_is_late_is_taken_when_its_proposal_arrives() {
let mut g = Game::new(Rules::default());
let red_home = Order::Take(g.tower("Red home"));
let mut c = bare(&g).with_advisor(Fixed(proposal(0, red_home, 5.0)), 3.0);
let mut c = bare(&g).with_advisor(Fixed(proposal(0, red_home, 8.0)), 3.0);
for _ in 0..3 {
c.decide(&g.view(Team::Blue));
assert_ne!(c.order(0), Some(red_home), "at {}", g.time);
Expand All @@ -843,11 +844,11 @@ mod tests {
fn an_advisor_with_no_delay_is_taken_at_once_and_one_that_is_later_than_its_proposal_lasts_is_not() {
let mut g = Game::new(Rules::default());
let red_home = Order::Take(g.tower("Red home"));
let mut now = bare(&g).with_advisor(Fixed(proposal(0, red_home, 5.0)), 0.0);
let mut now = bare(&g).with_advisor(Fixed(proposal(0, red_home, 8.0)), 0.0);
now.decide(&g.view(Team::Blue));
assert_eq!(now.order(0), Some(red_home));
// Good for 5 s, and 8 s late: stale when it arrives.
let mut late = bare(&g).with_advisor(Fixed(proposal(0, red_home, 5.0)), 8.0);
let mut late = bare(&g).with_advisor(Fixed(proposal(0, red_home, 8.0)), 8.0);
for _ in 0..30 {
late.decide(&g.view(Team::Blue));
assert_ne!(late.order(0), Some(red_home), "at {}", g.time);
Expand Down
16 changes: 8 additions & 8 deletions domains/recreation/towers/tests/acceptance.rs
Original file line number Diff line number Diff line change
Expand Up @@ -618,10 +618,10 @@ fn bindings() -> Bindings<World> {
.arrange("the proposal in the hand of the Blue commander", |w, v| {
w.proposed = v.as_str().unwrap_or("none").to_string();
match v.as_str() {
Some("the Red home tower for 5 points") => w.propose(5.0, 60.0, 0.0),
Some("the Red home tower for 1 point") => w.propose(1.0, 60.0, 0.0),
Some("the Red home tower for 5 points, good for 5 seconds") => w.propose(5.0, 5.0, 0.0),
Some("the Red home tower for 5 points, from the map of 3 seconds before") => w.propose(5.0, 60.0, 3.0),
Some("the Red home tower for 8 points") => w.propose(8.0, 60.0, 0.0),
Some("the Red home tower for 4 points") => w.propose(4.0, 60.0, 0.0),
Some("the Red home tower for 8 points, good for 5 seconds") => w.propose(8.0, 5.0, 0.0),
Some("the Red home tower for 8 points, from the map of 3 seconds before") => w.propose(8.0, 60.0, 3.0),
_ => {}
}
})
Expand Down Expand Up @@ -998,23 +998,23 @@ mod advice {
#[test]
fn a_proposal_that_gains_enough_is_taken() {
let mut w = World::new();
w.propose(5.0, 60.0, 0.0);
w.propose(8.0, 60.0, 0.0);
w.blue_decides();
assert_eq!(w.blue.order(0), red_home(&w));
}

#[test]
fn a_proposal_that_gains_little_is_not_taken() {
let mut w = World::new();
w.propose(1.0, 60.0, 0.0);
w.propose(4.0, 60.0, 0.0);
w.blue_decides();
assert_eq!(w.say(w.blue.order(0)), "take the west tower");
}

#[test]
fn a_stale_proposal_is_not_taken() {
let mut w = World::new();
w.propose(5.0, 5.0, 0.0);
w.propose(8.0, 5.0, 0.0);
w.run(10.0);
w.blue_decides();
assert_ne!(w.blue.order(0), red_home(&w));
Expand All @@ -1027,7 +1027,7 @@ mod advice {
w.blue_holds(2);
w.hold("west", &Value::from("neutral"));
w.hold("Red home", &Value::from("held by Blue"));
w.propose(5.0, 60.0, 3.0);
w.propose(8.0, 60.0, 3.0);
w.blue_decides();
assert_eq!(w.say(w.blue.order(0)), "take the west tower");
}
Expand Down
2 changes: 1 addition & 1 deletion expectations/09-towers/rules/05-commander.expect
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Feature: The computer commander

The commander also thinks ahead. Every 5 s it plays the game forward 45 s in a small model.
The enemy drones in the model are guesses from its belief. It tries to send each free drone to
each other tower. It makes the change that gives most points, if the gain is more than 2
each other tower. It makes the change that gives most points, if the gain is more than 5
points. Thus it does not send a drone to a tower where the enemy has more drones than the
team can bring. This is the first [[advisor]] of the commander: [[Advice for the commander]]
tells how a commander takes advice, and what it does with no advisor.
Expand Down
14 changes: 7 additions & 7 deletions expectations/09-towers/rules/18-advice.expect
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Feature: Advice for the commander
to gain by it, and the time after which it is stale. The advisor does not give orders. The
commander decides.

The computer commander takes a proposal when it gains more than 2 points, when it is not
The computer commander takes a proposal when it gains more than 5 points, when it is not
stale, and when it still fits: the drone is free to take an order, and the tower is not one
that the team holds now. Of the proposals that it can take, it takes the one that gains most.
A proposal that does not fit is dropped.
Expand All @@ -22,17 +22,17 @@ Feature: Advice for the commander
not take it.

Expectation: A proposal that gains enough is taken
The advisor proposes that the first Blue drone takes the Red home tower, for 5 points. At the
The advisor proposes that the first Blue drone takes the Red home tower, for 8 points. At the
start the commander sends the first drone to the west tower.

Given the proposal in the hand of the Blue commander is the Red home tower for 5 points
Given the proposal in the hand of the Blue commander is the Red home tower for 8 points
When the computer commander of Blue weighs the proposal
Then the order of the first Blue drone is take the Red home tower

Expectation: A proposal that gains little is not taken
The commander takes a proposal only when it gains more than 2 points.
The commander takes a proposal only when it gains more than 5 points.

Given the proposal in the hand of the Blue commander is the Red home tower for 1 point
Given the proposal in the hand of the Blue commander is the Red home tower for 4 points
When the computer commander of Blue weighs the proposal
Then the order of the first Blue drone is take the west tower

Expand All @@ -42,7 +42,7 @@ Feature: Advice for the commander
(The lookahead used to send it to the centre tower here; its model has the flight now, and
leaves the west tower to it.)

Given the proposal in the hand of the Blue commander is the Red home tower for 5 points, good for 5 seconds
Given the proposal in the hand of the Blue commander is the Red home tower for 8 points, good for 5 seconds
When the computer commander of Blue weighs the proposal 10 seconds later
Then the order of the first Blue drone is take the west tower

Expand All @@ -53,7 +53,7 @@ Feature: Advice for the commander
Given the towers held by Blue are 2
And the west tower is neutral
And the Red home tower is held by Blue
And the proposal in the hand of the Blue commander is the Red home tower for 5 points, from the map of 3 seconds before
And the proposal in the hand of the Blue commander is the Red home tower for 8 points, from the map of 3 seconds before
When the computer commander of Blue weighs the proposal
Then the order of the first Blue drone is take the west tower

Expand Down
10 changes: 5 additions & 5 deletions expectations/09-towers/section.expect
Original file line number Diff line number Diff line change
Expand Up @@ -272,11 +272,11 @@ Vocabulary:
Shows: the team is behind: the app shows an ADVISOR line with the score and the time left
Shows: none: the ADVISOR line says nothing new
# The advisors of the computer commander (recreation_towers::advice): proposals that it takes or not.
State: the proposal in the hand of the Blue commander is none | the Red home tower for 5 points | the Red home tower for 1 point | the Red home tower for 5 points, good for 5 seconds | the Red home tower for 5 points, from the map of 3 seconds before
Shows: the Red home tower for 5 points: an advisor proposes that the first Blue drone takes the Red home tower, for a gain of 5 points
Shows: the Red home tower for 1 point: an advisor proposes the same, for a gain of 1 point
Shows: the Red home tower for 5 points, good for 5 seconds: an advisor proposes the same, for a gain of 5 points, good for 5 seconds
Shows: the Red home tower for 5 points, from the map of 3 seconds before: an advisor proposes the same, for a gain of 5 points, from the map as it was 3 seconds before
State: the proposal in the hand of the Blue commander is none | the Red home tower for 8 points | the Red home tower for 4 points | the Red home tower for 8 points, good for 5 seconds | the Red home tower for 8 points, from the map of 3 seconds before
Shows: the Red home tower for 8 points: an advisor proposes that the first Blue drone takes the Red home tower, for a gain of 8 points
Shows: the Red home tower for 4 points: an advisor proposes the same, for a gain of 4 points
Shows: the Red home tower for 8 points, good for 5 seconds: an advisor proposes the same, for a gain of 8 points, good for 5 seconds
Shows: the Red home tower for 8 points, from the map of 3 seconds before: an advisor proposes the same, for a gain of 8 points, from the map as it was 3 seconds before
Event: the computer commander of Blue weighs the proposal
Say: wait for the next decision of the computer commander, which weighs the proposal in its hand
Event: the computer commander of Blue weighs the proposal 10 seconds later
Expand Down
2 changes: 1 addition & 1 deletion expectations/glossary.expect
Original file line number Diff line number Diff line change
Expand Up @@ -406,7 +406,7 @@ Term: Advisor
Term: Proposal
In [[Towers]], what an [[advisor]] sends to a [[commander]]: one order for a drone, the
[[queue]] that goes with it, the points that the advisor expects to gain by it, and the time
after which it is stale. A computer commander takes a proposal that gains more than 2 points
after which it is stale. A computer commander takes a proposal that gains more than 5 points
and is not stale. A person takes a proposal with a click on its line in the ADVISOR panel, or
with the key Y.

Expand Down
27 changes: 24 additions & 3 deletions plans/actors.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ the referee's view, and its ground sends the true places of the enemy as traffic
An advisor gets the team's map and gives **proposals**: a unit, an instruction, the gain it
expects, and the time until the proposal is stale. It runs beside the commander, not in its loop.
A person sees proposals in the ADVISOR panel and accepts one with a click. A computer commander
takes advice by itself: it takes a proposal whose gain is above a threshold (the lookahead's rule today: more than 2
takes advice by itself: it takes a proposal whose gain is above a threshold (the lookahead's rule today: more than 5
points). In the fast arena an advisor's delay is counted in game time, so a game stays
reproducible.

Expand Down Expand Up @@ -156,7 +156,7 @@ server.
map does not show: its orders after its own rules, the drones it may still change and the towers
each must stay away from; a seat has it, for the seat is the commander. A `Commander` holds
advisors, each with a delay in game time (`with_advisor(advisor, delay)`), and takes the best
proposal each decision that is not stale (`until`), gains more than `ADVICE_MARGIN` (2 points, a
proposal each decision that is not stale (`until`), gains more than `ADVICE_MARGIN` (5 points since the fresh-seed test below; 2 before, a
constant: `CommanderTuning` is the league's candidate format and stays as it is) and still fits
(the drone is free and the tower is not one it gave up, nor, for a proposal made on an earlier map,
one the team holds now). `Commander::advise` puts proposals in hand from outside, for a person's
Expand Down Expand Up @@ -275,7 +275,7 @@ server.
What it comes to. The model's missing flight was the defect: with it the lookahead goes from -0.47
to +0.05 ± 0.04 (430 won, 370 lost), which is level with no advisor, not a gain. A larger margin
(5 points) wins by +0.18 ± 0.04; that is a number found by looking at the result, and is not in the
code (`ADVICE_MARGIN` is 2 as before). Not fixed, and for the owner: whether the default commander
code (`ADVICE_MARGIN` is 2 as before; tested on fresh seeds below). Not fixed, and for the owner: whether the default commander
keeps the lookahead. With the fix `judge --game captain --champion timid --challenger default
--seeds 24` gives the default commander +0.384 (lower bound +0.048, promoted) with the lookahead,
+0.268 (+0.031) with `SIM001_LOOKAHEAD=0`, and +0.175 (-0.132, kept out) with the lookahead as it was.
Expand All @@ -293,6 +293,27 @@ server.
drones still lose (-6.4 ± 1.5 a move with the flight model); why drift and redeploy, which are the
referee's and the commander's own rules, measure worse than leaving them out (about 2 sigma each, not
checked at more seeds); why 20 s and 40 s of cooldown per drone are so much worse than 10 s.

**Margin on fresh seeds.** Rule written before the run: adopt 5 only if, on seeds 10001..=10800
(nobody had looked at them; 1600 games a row, both sides), a commander with margin 5 beats one with
margin 2 head to head, paired, with a 95 % lower bound (margin - 1.96 error) above 0; do not pick
another number from the table. `cargo run --release -p cognition_towers --example margin 800 10000`:

| Margin | Against no advisor | Won | Lost | Against margin 2 (lower bound) | Won | Lost |
| --- | --- | --- | --- | --- | --- | --- |
| 2 | -0.004 ± 0.031 | 833 | 767 | +0.000 ± 0.031 (-0.060) | 799 | 799 |
| 3 | -0.031 ± 0.031 | 814 | 786 | +0.074 ± 0.030 (+0.015) | 841 | 758 |
| 5 | +0.159 ± 0.032 | 916 | 683 | +0.245 ± 0.030 (**+0.187**) | 945 | 653 |
| 7 | +0.103 ± 0.033 | 884 | 716 | +0.076 ± 0.030 (+0.017) | 836 | 763 |
| 10 | +0.172 ± 0.033 | 923 | 677 | +0.064 ± 0.030 (+0.005) | 829 | 770 |

Verdict: the rule passes, and `ADVICE_MARGIN` is 5. Margin 10 is level with 5 against no advisor
(+0.172 against +0.159, well inside the error) and was not tested head to head against 5: a lead
for another fresh test, not a change. After (`advice 400`, margin 5; before in brackets):
0 s +0.19 ± 0.04 (+0.05), 2 s +0.03 (-0.16), 5 s -0.09 (-0.27), 10 s -0.19 (-0.55), 30 s -0.84
(-0.94); 0 s: 468 won, 332 lost (430, 370). `judge --game captain --champion timid --challenger
default --seeds 24`: +0.453, lower bound +0.215, promoted (+0.384, +0.048 before); 32 won, 16 lost
(30, 18). Both before numbers were re-run on this master and equal the ones above.
**Results (the seat side: advisors beside the commander, a process advisor, a Claude advisor,
and the app's accept).** Done in `recreation_towers_ground::{advisors, claude}`, the commander
app and the server. An advisor at a seat runs on a thread of its own (`advisors::Runner`) behind
Expand Down
Loading