Skip to main content

Course D — Robot Town: Amending Rules Toward a KPI

⬅ Back to Course A → Course A — Robot Delivery

Audience: Students who have completed Course A (CADL/SoS-DSL basics). Course B is not required.

Time: 120–150 minutes, plus optional stretch goals.

You will leave with: KPIs written by a human in natural language, the LLM's amendment proposals and decision requests, the human's decisions with their reasons (in prose and in GSN), and the versions of the CADL rules and of the constitution that resulted, period by period.


0. What this course demonstrates​

Courses A and B write a contract once and check a simulation against it. In Course D the contract itself changes while the system runs. Three delivery companies (A, B, C; two robots each) share the 11-node road network of the Raspberry Pi Mouse simulator. After each simulated hour (one period), amendments to the rules are considered.

A human, an LLM and a set of mechanical gates share the work:

WhoDoes what
Human (the service operator)Writes the service KPIs in natural language; the KPIs may change over time. Answers the LLM's decision requests and records the reason and the outcome, in prose or in GSN
LLMTurns the KPI into measurable metrics and proposes amendments to rules, regulation and contracts toward it. When a question cannot clearly be settled, or when human stakeholders must agree, it does not decide: it asks the human and supplies what the decision needs
GatesCheck, constitution, improvement measured on the same random seeds, and the companies' vote. The LLM's own account is never trusted; only measured values count

The exercise lives in the public repository cadl-robot-town. It needs neither Unity nor SUMO: the town is a small Python simulation, and the norms are judged by the same reference runtime as Course A (contract_engine.py, copied from cadl-raspimouse-simulator).

About the recorded LLM output

§4–§8 can be followed without an API key by replaying the records in examples/steward-d-en/ (English) and examples/steward-goodhart/. They are records of real runs with Claude Opus 5.5 on 2026-11-16; examples/steward-d/ holds a run in Japanese with the same KPI, which parts from the English one in period 1 (§10). The answers to the decision requests (answers.yaml) were written by Claude playing the human operator when the records were made; from §4 on, answer them yourself. Every number shown is measured by the gates and the simulation on replay. §10 shows how to run a real LLM with your own key.


1. Setup (5 min)​

All you need is the CADL compiler cloned in Course A (~/program/cadl_repo).

cd ~/program
git clone https://github.com/ertlnagoya/cadl-robot-town
cd cadl-robot-town
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
pip install -e ~/program/cadl_repo # the cadl command and PyYAML

If the install fails, see the setup troubleshooting in Section 2.4 of the Textbook.

Only §10 (a real LLM) needs the Anthropic SDK and an API key.

pip install anthropic

town.steward --lang en prints English and asks the LLM to write in English. town.period, town.council and town.compare print Japanese only; each step below explains what you need from their output.


2. Read the rules, the contract and the regulation (15 min)​

Open cadl/robot_town.cadl. The town has two contracts.

  • DELIVERY_SLA: one instance per order. Pick up within 120 s and deliver within 120 s of pickup, or the order becomes Late. This is the promise to customers.
  • ROAD_USE: one instance each time a robot stops on the road to wait for its next order.
- id: DELIVERY_SLA
parties: [TOWN_STEWARD, "ROBOT[*]"]
guarantee:
- "pickup_time <= 120s" # @amend:pickup_deadline
- "dropoff_time <= 120s" # @amend:dropoff_deadline
lifecycle:
...
- id: pickup
from: Assigned
to: PickedUp
on: "ROBOT[i].status == PickedUp"
deadline: 120s # @amend:pickup_deadline
on_violation: { transition: Late, severity: Major }
...
- id: ROAD_USE
parties: [TOWN_STEWARD, "ROBOT[*]", COUNCIL]
authority:
decision_holder: COUNCIL
incentives:
type: sanction
lambda: 0.5 # @amend:sanction_lambda
lifecycle:
states: [Holding, Overstay, Released, Evicted]
initial: Holding
terminal: [Released, Evicted]
transitions:
- id: release
from: [Holding, Overstay]
to: Released
on: "ROBOT[i] -> TOWN_STEWARD : node_released"
deadline: 60s # @amend:hold_limit
on_violation: { transition: Overstay, severity: Minor }
monitors:
- id: blocking_guard
observe: "ROBOT[i].blocking"
sampling: periodic(1000ms)
rule: "ROBOT.blocking >= 3 AND state IN [Holding, Overstay]" # @amend:block_k
on_match: { transition: Evicted, severity: Major }

(Excerpt, simplified.)

The two lifecycles, as state machines:

A line tagged # @amend:<name> holds a parameter that may be amended. When several lines carry the same name (the pickup guarantee and the pickup deadline), they must all hold the same value. Which layer a parameter belongs to and how far it may move is set by constitution.yaml:

amendable:
hold_limit: {layer: rule, min: 10, max: 120, step: 10} # seconds a robot may wait on the road
block_k: {layer: rule, min: 1, max: 4, step: 1} # robots waiting behind it before eviction
sanction_lambda: {layer: rule, min: 0.0, max: 1.0, step: 0.5} # sanction points per violation (Minor 1, Major 3)
to_central: {layer: rule, safety: true, ...} # operating-mode switch (§9)
pickup_deadline: {layer: contract, min: 60, max: 180, step: 30} # promised seconds to pickup
dropoff_deadline: {layer: contract, min: 60, max: 180, step: 30} # promised seconds to delivery
invariants:
min_company_on_time_rate: 0.6
vote:
quorum: 2
regulation:
invariants.min_company_on_time_rate: {min: 0.5, max: 0.95, step: 0.05}
vote.quorum: {min: 1, max: 3, step: 1}

(Excerpt, simplified.)

There are three layers of amendment:

LayerCoversWho decides
RuleROAD_USE and the operating-mode switching conditionsadopted automatically if it passes every gate; the switching conditions (safety: true) are decided by the human
ContractDELIVERY_SLA (the promise to customers)customers have no vote, so the human decides
Regulationvalues in the constitution (invariants, quorum)the human decides

The three companies behave differently (town/policies.py): A always returns to its garage; B waits on the road for its next order but leaves 5 s before hold_limit, and does not wait on the road at all if a single waiting robot triggers eviction (block_k of 1); with no sanctions (sanction_lambda of 0) a violation costs it nothing, so it waits as long as C does. C waits until an order comes or it is evicted; at the highest sanction weight (sanction_lambda of 1.0) it leaves 15 s after the time limit. CADL's check keeps the sanction weight between 0 and 1, and an amendment moves it in steps of 0.5; §3 shows the companies at 0.5. A robot stopped at an intersection blocks everyone who wants to pass through it.

The road network has 11 nodes and 17 roads (town/world.py). The garages of A, B and C are nodes 10, 9 and 4 (yellow). A robot that waits on the road waits at the central intersection 8 (purple), the most convenient place to wait.

Road network of the robot town: the outer ring of nodes 0 to 7, the central intersection 8, and the garages 10 (A), 9 (B) and 4 (C)

Exercise 2-1​

  1. Which company can never be sanctioned under the current rules, and why?
  2. If the pickup promise (pickup_deadline) were extended to 150 s, what would you expect to happen to the number of on-time deliveries? And to how long customers actually wait?
  3. Why must an amendment to the contract layer not be settled by the companies' vote alone?

3. Run one period (10 min)​

Fix the operating mode to distributed for now (§9 explains the modes):

python -m town.period --mode DISTRIBUTED
需要: 各社の注文の平均間隔 75 秒
運用モード: DISTRIBUTED
規則: robot_town.cadl (hold_limit=60s, block_k=3, sanction_lambda=0.5)
定時配達 121.8 / 注文 132.2 未配達 0.0 ふさがれた時間 2404s (シード平均)
会社 定時率 遅配 道路で待った回数 ふさがれた秒数 罰点 効用
A 0.84 7.8 0.0 1352 0.0 38.4
B 0.94 2.6 38.0 849 0.9 38.9
C 1.00 0.0 39.6 203 14.8 28.8
違反(原因別,シード平均): deadline:dropoff=2.6, deadline:pickup=7.8, deadline:release=23.0, monitor:blocking_guard=2.8
運用モード: 集中の時間 0% 切替 0.0 回(切替中の合計 0s,管理者による退去 0.0 台,安全条件の違反 0.0 回)

The columns are: on-time rate, late deliveries, times waited on the road, seconds blocked by stopped robots, sanction points, and utility (on-time deliveries minus sanction points). The numbers are averages over five random seeds.

C waits on the road, reaches every customer on time, and pays 14.8 sanction points for it; A, which never blocks anyone, is blocked for 1352 s and loses 7.8 deliveries. Every violation in the second-to-last line was detected by the contract runtime from the IR: deadline:* by lifecycle deadlines, monitor:blocking_guard by the monitor.


4. Write the KPI (15 min)​

The human's first job is to write the KPI. In English it goes in kpi.en.md:

## From period 1

Deliver as many orders as possible across the town within the times we promised customers
(120 seconds from order to pickup, 120 seconds from pickup to delivery). No company's customers
should be kept waiting badly.

## From period 5

The quiet season begins and orders fall. On-time delivery still matters, but to keep the
companies cooperating, no single company should gain or lose because of how the road is used.

A ## From period k heading says when a KPI takes effect (the Japanese file, kpi.md, uses ## 第 k 期から). The loop re-reads the file at the start of every period, so you may edit it between periods. --lang en makes the loop's own text and the LLM's writing English.

A natural-language KPI cannot rank proposals by itself. The LLM rewrites it as a combination of the metrics in town/kpi.py (on-time deliveries, late deliveries, unfinished orders, blocked time, the lowest company's on-time rate, the spread of the companies' utilities, total sanctions, and so on). This is the measure: a main goal (improvement is measured by it), watched goals (the human is told if they get worse by more than the noise) and conditions (which every amendment must keep).

Replay the record and run one period. Without --answers, you answer decision requests in the terminal.

python -m town.steward --lang en --mode DISTRIBUTED --rounds 1 --name try-d-en \
--llm replay --replay-dir examples/steward-d-en
KPI (from period 1): Deliver as many orders as possible across the town within the times we promised customers (120 seconds from order to pickup, 120 seconds from pickup to delivery). No company's customers should be kept waiting badly.
LLM: replay: examples/steward-d-en/p1-kpi.json (claude-opus-5-5, input 2431 / output 2306 tokens)
Reading: The KPI has two parts. The main objective is the town-wide count of deliveries that meet both promised windows (pickup within 120 s of the order, delivery within 120 s of pickup), which maps to on_time_deliveries. The second sentence is a fairness floor: no single company's customers should get badly worse service, which maps most directly to min_company_on_time_rate. (…)
[1] Maximize town-wide on-time deliveries, with a floor on the worst company's on-time rate: main goal on_time_deliveries (increase), min_company_on_time_rate (increase), unfinished (decrease), late_deliveries (decrease); conditions min_company_on_time_rate >= 0.7
[2] Maximize the worst company's on-time rate, with town-wide volume as a monitored goal: main goal min_company_on_time_rate (increase), on_time_deliveries (increase), unfinished (decrease); conditions on_time_rate_gap <= 0.2
[3] Maximize town-wide on-time deliveries; read 'kept waiting badly' as late or abandoned orders: main goal on_time_deliveries (increase), late_deliveries (decrease), min_company_on_time_rate (increase); conditions unfinished <= 0
Concern: 'Kept waiting badly' is not quantified. It could mean a per-company on-time rate floor, a cap on the rate gap between companies, or no abandoned or very late orders. The guard thresholds (0.7, 0.2, 0) are placeholders the human must confirm.
Concern: on_time_deliveries is a count. Optimizing it alone can reward rules that let the hard or remote orders of one company go unfinished or late, which goes against the fairness sentence. (…)
Concern: Both metrics depend on the 120 s windows in DELIVERY_SLA. Relaxing the SLA would inflate on-time counts without real improvement, so the SLA must be treated as fixed under this KPI. Any change to it is a contract decision for the human.
(2 more)
→ Decision request K1 (the LLM recommends 1)
Choose an id (1 / 2 / 3):

This replays what Claude Opus 5.5 actually wrote on 2026-11-16. When a KPI has a single reading and no concern, the LLM uses its measure without asking. Here it found three readings of "kept waiting badly" and returned three measures with five concerns. The machine checks the metric names and the shape of every measure the LLM writes; one that uses an unknown metric is dropped from the choices.

After you pick a number you are asked for a reason and for assumptions (GSN Assumptions; any number of lines, an empty line ends). The recorded answers (examples/steward-d-en/answers.yaml) were written by Claude playing the operator when the record was made:

K1:
choice: "1"
reason: Follow the order of the KPI. Town-wide on-time deliveries are the main goal, and the worst company's on-time rate has a floor. The floor of 0.7 is provisional and will be revisited once period 1 is measured.
assumptions:
- Customers cannot choose which company delivers their order

Exercise 4-1​

  1. How do you expect the decision request in §5 to change if you pick measure 2?
  2. The last concern shown restates question 2 of Exercise 2-1. Where in the KPI's wording did the LLM find that?

5. Answer a decision request (20 min)​

After K1, the LLM proposes amendments and each passes through the gates:

| id | proposal | layer | check | constitution | on-time | late | blocked s | road violations | lowest on-time rate | change in on_time_deliveries | seeds improved | KPI conditions | utility A / B / C | votes A B C |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| — | current | — | — | — | 121.8 | 10.4 | 2404 | 25.8 | 0.84 | — | — | — | 38.4 / 38.9 / 28.8 | — |
| 1 | hold_limit 60s → 50s | rule | ○ | ○ | 120.6 | 11.6 | 2468 | 28.8 | 0.86 | -1.2 | 2/5 | ○ | 38.8 / 37.6 / 27.2 | ○ × × |
| 2 | block_k 3 → 2 | rule | ○ | ○ | 131.0 | 1.2 | 985 | 34.0 | 0.98 | +9.2 | 5/5 | ○ | 45.4 / 37.2 / 19.0 | ○ × × |
| R1 | invariants.min_company_on_time_rate 0.6 → 0.65 | regulation | — | ○ | | | | | | | | | | |

(The "watched goals and conditions" and "notes" columns are left out here; the note on R1 reads "the current rules meet the new condition".) The LLM's second proposal, block_k 3 → 2 (evict a robot as soon as two robots wait behind it), adds 9.2 on-time deliveries but is voted down by B and C. It also proposed a regulation: raising the constitution's floor on every company's on-time rate from 0.6 to 0.65. It proposed no contract change, having read at K1 that loosening the promised times does not count as improving the KPI. It did not propose a heavier sanction weight either; the Japanese run did, in the same period (§10).

The loop does not decide when a proposal raises any of the following (triage() in town/steward.py):

ReasonWhen
Agreementit improves the KPI but is voted down, and beats every proposal that could be adopted automatically
Contractit changes the promise to customers and improves the KPI
Regulationit changes a value of the constitution
Safetyit changes an operating-mode switching condition and improves the KPI
KPI conflictit improves the main goal but breaks a KPI condition or worsens a watched goal
Uncertainit passes every gate but improves on fewer than 4 of the 5 seeds
LLMthe LLM itself said a human should decide (only for proposals that keep the constitution's invariants)

Each proposal passes through the gates in this order before the triage:

Here proposal 2 raises agreement and R1 raises regulation. The LLM writes decision request D1, saved as runs/try-d-en/p1-request.md:

→ Decision request D1 (LLM: replay: examples/steward-d-en/p1-request.json (claude-opus-5-5, input 2512 / output 2072 tokens))
**Question**: Should we adopt the rule change block_k 3 → 2 even though companies B and C voted against it, raise the constitutional floor min_company_on_time_rate from 0.6 to 0.65, or keep the current rules (v1)?
- **2**: block_k 3 → 2. (…) Votes: A ○, B ×, C ×. (stakeholders: Gains: customers of all companies (far fewer late deliveries, and the worst-served company's on-time rate rises to 0.98) and company A (utility 38.4 → 45.4). Losses: company C (utility 28.8 → 19.0, a large drop), company B slightly (38.9 → 37.2), and possibly road safety and order, since road violations rise 25.8 → 34.0. Adopting it overrides the stated preference of two of the three companies.)
- **R1**: (…) The current rules already meet the new condition (current lowest on-time rate is 0.84), so nothing changes in operation now. It only tightens the limit on future rule changes. (…)
- **keep**: no change. (…)
The LLM recommends: 2 — It is the only option that moves the KPI. The main goal improves by +9.2 on 5/5 seeds, late deliveries drop from 10.4 to 1.2, and the KPI's fairness measure (lowest on-time rate) improves from 0.84 to 0.98, so no company's customers are worse served. However, it overrides B and C, C's utility falls sharply (28.8 → 19.0), and road violations rise (25.8 → 34.0). If you adopt it, consider recording why the override is justified and whether C should be compensated or the change reviewed after one period. (…)
The LLM does not know: Why B and C voted against proposal 2, and what drives C's utility drop despite better on-time performance (e.g., cost, route share, or priority loss).
The LLM does not know: How B and C will behave if overridden (strategic responses, reduced cooperation, or exit), which was not simulated.
(5 more)

A request contains the question; why the LLM is not deciding; for each option what happens and who gains or loses; the LLM's recommendation; and what the LLM does not know. The options are the proposals that raised a reason, the proposal that could be adopted automatically (if any) and "keep"; the LLM cannot add options. The table of numbers is the gates' measurement, not the LLM's prose.

The recorded answer follows the recommendation and adopts proposal 2, overturning the vote:

D1:
choice: "2"
reason: The intersections are a road shared by the three companies; waiting there is no company's right. Late deliveries fall from 10.4 to 1.2 and the worst company's on-time rate rises to 0.98, so adopt it against B's and C's votes. Re-measure C's utility (28.8 to 19.0) and the road violations next period. The constitutional floor (R1) can wait; it changes nothing now.
assumptions:
- B and C keep delivering in the town after the rules change

In the earlier records of this course (2026-11-15), the English LLM recommended R1 for the same proposals and numbers, and the human overturned the vote against that recommendation. In the records of 2026-10-29, the Japanese recommendation changed when the same request was sent a second time on the same numbers (first "1", then "keep"). A recommendation is the output of one request. The LLM laid out the material and showed that each choice had reasons; the human decided.

The record of decisions​

Each decision goes into two files in runs/<name>/:

  • decisions.md: the prose record — for each decision, the measure, the LLM's assessment, why the human was asked, the LLM's recommendation and unknowns, and the human's choice and reason.
  • gsn.yaml: a GSN (Goal Structuring Notation) tree, with a Mermaid diagram in gsn.md.
GSN elementWhat goes in it
Goalthe KPI written by the human (one per version)
Strategythe decision of a period (the amendment adopted, or no change)
Solutionthe measurement on the same seeds (p<k>-evidence.md)
Contextthe measure of the KPI; the period's environment
Assumptionassumptions written by the human
Justificationthe human's reason (or, for an automatic decision, its condition)

The tree in gsn.md has this shape. Solid arrows are supported by; dotted arrows are in context of.

Reasons may be written in prose or as GSN elements; in the --answers YAML, list elements (each with type: and text:) under gsn:. For example (the recorded answer to D2 gives its reason in prose):

D2:
choice: keep
gsn:
- type: justification
text: Only 2 of 5 seeds improve; do not cut C's utility by 16 for an uncertain gain
- type: context
text: In D1 we decided to re-measure C's utility

From the next period on, earlier decisions and the human's reasons are passed to the LLM. §6 shows how that changes its proposals and recommendations.

Compare: a council with no human​

The same first period with the rule-based council (town.council), which tries every one-step amendment to the rules and adopts the gate-passing one with the best objective, never a vetoed one:

python -m town.council --mode DISTRIBUTED
改正案 定時配達 違反 (道路) ふさがれた 検査 憲法 改善 投票(A B C)
hold_limit 60s → 50s 120.6 40.4 (28.8) 2468s ○ ○ × ○ × ×
hold_limit 60s → 70s 124.2 31.6 (23.6) 2367s ○ ○ ○ ○ ○ ○
block_k 3 → 2 131.0 35.2 (34.0) 985s ○ ○ ○ ○ × ×
block_k 3 → 4 119.4 36.4 (23.6) 2788s ○ ○ × ○ × ○
sanction_lambda 0.5 → 0 119.0 59.0 (45.8) 3059s ○ ○ × ○ ○ ○
sanction_lambda 0.5 → 1 132.2 23.6 (23.6) 845s ○ ○ ○ ○ ○ ×
...
→ 採択: sanction_lambda 0.5 → 1(目標 +10.4,賛成 AB)。v2.cadl として保存。

(town.council prints Japanese only.) The council tries every one-step amendment, including the heavier sanction weight the English LLM did not propose. At weight 1, C leaves the intersection 15 s after the time limit, blocked time falls from 2404 s to 845 s, and every late delivery disappears. C votes against, but A and B make the quorum of two. block_k 3 → 2 is vetoed by B and C. Switching off sanctions (0.5 → 0) wins all three votes, but B then waits as long as C does, on-time deliveries fall, and the improvement gate drops it. D1 reached the town's second-best rule by overturning a veto; the council reached a better one because it tried a proposal the LLM never wrote. The council keeps no reason, and the LLM's proposals depend on what it is told about the companies (§11).

Exercise 5-1​

  1. Answer keep (or R1) to D1 and rerun the first period. How do decisions.md and gsn.md change?
  2. B is itself blocked for 849 s, yet it votes against block_k 3 → 2. Explain why from the table in §3.

6. Run six periods (20 min)​

Run all six periods with the recorded answers:

python -m town.steward --lang en --mode DISTRIBUTED --rounds 6 --name steward-d-en \
--llm replay --replay-dir examples/steward-d-en --answers examples/steward-d-en/answers.yaml

scenario.yaml sets the demand of each period (normal, busy, quiet, with a mid-period rush in some). The current rules are re-measured in each period's environment before being compared.

PeriodReasons for the requestLLM recommendsHuman's choiceResult
1 (normal)agreement (block_k 3 → 2 vetoed by B and C); regulation (floor 0.6 → 0.65)2block_k 3 → 2, overturning the vetov2
2 (normal + rush)uncertain (block_k 2 → 1 improves on 2/5 seeds); LLM; regulation (floor 0.6 → 0.65)keepkeepv2
3 (busy)none (the only proposal, hold_limit 60s → 70s, gains +0.4, below the noise of one delivery)——v2 (automatic)
4 (busy + rush)none (the LLM proposes nothing)——v2 (automatic)
5 (quiet)see §73hold_limit 60s → 70sv3
6 (quiet)KPI conflict (hold_limit 70s → 80s lowers the lowest on-time rate from 0.97 to 0.94)1keepv3

The Japanese record (examples/steward-d/) parts from this in period 1. There the LLM also proposed doubling the sanction weight (sanction_lambda 0.5 → 1), which removed every late delivery and passed the vote with A and B; the human adopted it, and kept the rules in every later period. The Japanese rules end at v2 (see §10).

Things to notice:

  • Period 2: block_k 2 → 1 passes every gate, but only 2 of 5 seeds improve. The LLM also asked on its own, citing the human's promise in D1 to re-measure C's utility. It wrote that the change "will very likely fail the 2-of-3 vote, with B and C opposed, so adopting it would mean a second override"; in fact A and B vote for it, which reaches the quorum of 2. The decision-request call is not given the constitution, so the LLM did not know the quorum. It did list the mismatch as unknown. The recorded answer keeps the rules, noting that no override is needed.
  • Periods 3–4: in the busy season there are few late deliveries left, so there is little room for the main goal. The LLM says so in its assessment and proposes nothing in period 4. No request; the GSN still records that the decision was automatic, and on what condition.
  • Period 6: the LLM recommends hold_limit 70s → 80s as a continuation of D5: the gain is consistent (4/5 seeds), the 0.9 floor still holds (0.94), and B and C make the quorum. The recorded answer keeps the rules: the spread narrows only because A and B lose a little and C gains a little, while customers get slightly worse service.

Open runs/steward-d-en/gsn.md to see the tree of six periods of decisions (GitHub and the VS Code preview render it as a Mermaid diagram).

Seeing the versions of the rules in CADL Explorer​

The rule versions runs/steward-d-en/v1.cadl to v3.cadl can be compared on the Designer page of CADL Explorer (v0.5.9 or later).

  1. Open v1.cadl under Open an existing CADL file, switch the view to Versions, type v1 as the name and press Save as a version. Do the same for v2.cadl and v3.cadl.
  2. Under Amendable parameters, upload this repository's constitution.yaml as the Constitution (Use the Robot Town constitution loads the same file). The table lists each # @amend: name with its value, layer, range, step and decided by: the checks for the rule layer, a person for the contract layer and for the safety-related operating-mode rules — the same division as in §2.
  3. Open History of the amendable values across the versions. block_k changes from 3 to 2 in v2 and hold_limit from 60 to 70 in v3: the amendments of periods 1 and 5 in the table above.
  4. Under Compare two designs, choose v1 as A and v2 as B. The amended parameter (block_k 3 → 2) is listed above the structural changes and the source diff.

Explorer checks the values with the same rules as the constitution gate of this course: only numbers on # @amend: lines change, one name at a time, by one step, within the range. It does not run the other gates (the checks of cadl aside, it measures no improvement and holds no vote). Amending a value there (the Amend field) keeps the comments, but its decision record lasts only for the browser session unless you download it. The record of this course is decisions.md and gsn.yaml.

Exercise 6-1​

Copy examples/steward-d-en/answers.yaml, change D2 to "1", and pass the copy to --answers. The recorded LLM outputs assume the recorded answers; once an answer differs, every later replay prints "note: the request differs from the recorded one". At which gate is the LLM's period-3 proposal stopped, and why? What would happen with a real LLM (§10)?


7. The KPI changes (15 min)​

From period 5, the other section of the KPI file takes effect. When the KPI changes, the LLM rewrites the measure and lists earlier decisions whose premises no longer hold:

KPI (from period 5): The quiet season begins and orders fall. On-time delivery still matters, but to keep the companies cooperating, no single company should gain or lose because of how the road is used.
[1] Fairness of outcomes first: minimize the utility gap, with an on-time floor: main goal utility_gap (decrease), min_company_on_time_rate (increase), on_time_deliveries (increase), late_deliveries (decrease); conditions min_company_on_time_rate >= 0.9
[2] Fairness of service: minimize the on-time rate gap, with an on-time floor: main goal on_time_rate_gap (decrease), utility_gap (decrease), on_time_deliveries (increase), unfinished (decrease); conditions min_company_on_time_rate >= 0.9
[3] On-time first, with fairness as a hard limit: main goal on_time_deliveries (increase), utility_gap (decrease), min_company_on_time_rate (increase); conditions on_time_rate_gap <= 0.1, min_company_on_time_rate >= 0.9
Concern: The KPI can be read two ways. Fairness can be the objective ('to keep the companies cooperating, no single company should gain or lose'), or a constraint on continuing to maximize on-time delivery ('on-time delivery still matters'). (…)
Concern: Any gap metric can be improved by leveling down, making the better-off companies worse rather than the worse-off better. The on-time floor and the monitored goals are meant to block this. The floor value of 0.9 is my choice based on the 0.98 measured after D1, and the human should confirm it.
(2 more)
To review: K1 — K1 made town-wide on-time deliveries the main goal with a provisional 0.7 floor. The new KPI puts inter-company fairness first, so K1 is superseded. (…)
To review: D1 — D1 (block_k 3 → 2) was adopted against B's and C's votes because it raised town-wide on-time delivery. It cut C's utility from 28.8 to 19.0. The new KPI says no company should lose because of how the road is used, so D1's justification no longer holds by itself. (…)
(2 more)
→ Decision request K5 (the LLM recommends 1)

The recorded answer picks measure 1 (narrow the utility spread), keeping the LLM's floor of 0.9. This period's proposals:

idProposalOn-timeLateMain goal (utility spread)Seeds improvedUtility A / B / CVotes A B CLowest on-time rate
—current80.81.8——25.2 / 25.4 / 10.2—0.96
1block_k 2 → 374.87.8+7.54/523.0 / 23.6 / 15.9× × ○0.83
2sanction_lambda 0.5 → 076.06.6+11.24/523.2 / 27.2 / 25.6× ○ ○0.89
3hold_limit 60s → 70s80.22.4+1.74/525.0 / 24.7 / 11.5○ × ○0.95

Proposal 2 (switch off sanctions) narrows the spread most and reaches the quorum. But with no sanctions B also waits on the road as long as C does: late deliveries rise from 1.8 to 6.6, road violations from 26.0 to 45.8, and the lowest on-time rate falls to 0.89, below the KPI condition of 0.9 (KPI conflict). Proposal 1 (undoing D1) is vetoed by A and B (agreement) and breaks the same condition. Proposal 3 passes every gate and could be adopted automatically. The LLM recommends 3:

The LLM recommends: 3 — It is the only option that narrows utility_gap without breaking the 0.9 floor you set in K5 to stop the gap from being narrowed by making service worse. It reaches the vote quorum, does not reverse D1, and does not reward overstays. Options 1 and 2 narrow the gap mainly by worsening service and either override two companies or remove the cost of blocking. (…)

In the earlier records of this course (2026-11-15), the robots did not respond to the sanction weight, so switching off sanctions changed no delivery number, narrowed the spread by 12.0, and the LLM recommended it. Whether a proposal is a real fix or only moves a number depends on how the companies behave, which the LLM does not see.

The recorded answer adopts 3:

D5:
choice: "3"
reason: It is the only option that narrows the utility spread while keeping the 0.9 floor, and A and C make the quorum. Reverting D1 or removing sanctions narrows the spread by making service worse (lowest on-time rate 0.83 and 0.89), which the floor was set to prevent.

Exercise 7-1​

  1. Pick measure 3 at K5 (copy answers.yaml and edit K5) and run period 5. How do the options of the decision request change? The recorded request was written for measure 1; what happens to it?
  2. Under which measure, and with which votes, would switching off sanctions be adopted automatically without a request? Is that desirable? In a replay the KPI reading is the recorded one, so to try a rewritten KPI file you need a real LLM (§10).

8. A KPI that is a proxy (15 min)​

What happens when the KPI aims at a convenient stand-in (a proxy)? examples/steward-goodhart/kpi.en.md says:

## From period 1

Make sure the road rules are kept. Reduce road-rule violations as much as possible.
python -m town.steward --lang en --kpi examples/steward-goodhart/kpi.en.md --mode DISTRIBUTED --name goodhart-en \
--llm replay --replay-dir examples/steward-goodhart/en --answers examples/steward-goodhart/en/answers.yaml

The LLM warned that "road_violations is a proxy that can be gamed at the rule layer. A ROAD_USE amendment that lengthens release deadlines or relaxes the blocking_guard would reduce counted violations without the road actually becoming better used", and offered three measures: (1) the count as main goal, watching blocked time and service (recommended); (2) zero violations as a hard condition; (3) blocked time as main goal, with the count watched. The recorded answer picks 1. The LLM proposed a single amendment, sanction_lambda 0.5 → 1. It cuts the count from 25.8 to 23.6 on 4 of 5 seeds, blocked time from 2404 s to 845 s and late deliveries from 10.4 to 0; A and B vote for it, and it is adopted automatically.

In Japanese (examples/steward-goodhart/proxy/ and guarded/), the record answers K1 in two ways: proxy takes the count as main goal and watches service and blocked time; guarded also watches the spread of the companies' utilities. In both, the LLM proposed the same three amendments:

idProposal (Japanese record)On-timeLateBlocked sRoad violationsChange in main goalSeeds improvedVotes A B C
—current121.810.4240425.8———
1sanction_lambda 0.5 → 1132.20.084523.6+2.24/5○ ○ ×
2block_k 3 → 2131.01.298534.0−8.20/5○ × ×
3hold_limit 60s → 70s124.28.0236723.6+2.25/5○ ○ ○

Proposals 1 and 3 cut the count to the same 23.6. Proposal 1 does it by making C leave the intersection sooner, and blocked time and late deliveries fall with it; proposal 3 does it by moving the line that counts as a violation (a robot may stop for 70 s instead of 60 s), and blocked time barely changes. The number alone cannot tell them apart. Proposal 2 tightens the rules too, but evicting earlier counts more evictions (−8.2). In both records the LLM sent proposal 3 to the human, writing (in Japanese) that it lowers the count by loosening the rule rather than by changing how the robots behave, and recommended 1. Under guarded, proposal 1 also raised a KPI conflict: only C's sanctions grow, so the watched spread of utilities widens from 10.1 to 27.0. Both recorded answers adopt 1.

In the earlier records of this course (2026-11-15), the robots did not respond to the sanction weight, so sanction_lambda 0.5 → 1 left the count unchanged and the only amendment that cut it was the loosening. Whether a proxy KPI pulls toward loosening depends on whether a real fix that also moves the number is on offer.

How far the drift goes with no reader of the KPI's intent can be seen with the rule-based council, which looks only at the objective's number:

python -m town.council --mode DISTRIBUTED --rounds 8 --objective road_violations --name goodhart-d
python -m town.council --mode DISTRIBUTED --rounds 6 --name seasons-d
python -m town.compare runs/goodhart-d/v1.cadl runs/goodhart-d/v5.cadl runs/seasons-d/v4.cadl --mode DISTRIBUTED
RulesOn-timeLateRoad-rule violationsBlocked timeA's on-time rate
initial v1112.210.825.02457 s0.82
v5, objective: road violations106.017.021.03681 s0.72
v4, objective: on-time deliveries122.80.219.8956 s1.00

(Compared on seeds 1001–1005, which the loop never used.) With road violations as its objective, the council amended the rules four times in eight periods, every time loosening them. The counted violations fell, while blocked time and late deliveries both rose by about half. The constitution's invariant (on-time rate at least 0.6) does not stop the drift. The heavier sanction weight was on the list every period and cut the count too, but a loosening always cut it as much or more (in period 1 both reached 23.6, and the loosening comes first in the list), and C always votes against heavier sanctions. The council with the on-time objective doubled the sanctions in period 1, and its rules end with fewer road violations than the ones that chased the count.

The council does not tell a loosening from a real fix when the numbers tie; in town.steward the LLM flagged the loosening and the human chose the fix. There is no guarantee that an LLM will read the KPI's intent as it did here. Choosing the measure is the human's job, and so is the decision when a loosening is proposed.


9. Distributed and centralized operation (15 min)​

ModeWhere robots waitHow the road is used
DISTRIBUTEDeach company on the road (the central intersection 8)intersections are free to pass
CENTRALIZEDa steward assigns off-road bays (beside nodes 5, 2 and 7, one robot each); waiting on the road is forbiddenthe steward grants each edge to one robot at a time (as the C-SoS arbiter does); entering an edge costs a 1 s grant

Centralized operation removes blocked intersections, but at very high demand the edge grants pile up. On-time deliveries with the mode fixed (mean over seeds 1–5):

Mean order interval30 s45 s75 s120 s180 s
Distributed319.6210.2121.876.851.2
Centralized309.2217.4132.283.258.6

The switching conditions are written in the CADL file and evaluated every 5 minutes over the last 5 minutes:

transitions:
- from: DISTRIBUTED
to: CENTRALIZED
condition: "orders_in_window < 22" # @amend:to_central
protocol: SWITCH
safety_invariant: "robots_on_road == 0"
- from: CENTRALIZED
to: DISTRIBUTED
condition: "orders_in_window >= 26" # @amend:to_distributed
protocol: SWITCH

A switch to centralized takes effect only once no robot is waiting on the road (C stalls for 30 s; anything still there after 60 s is evicted by the steward), and the safety_invariant is checked at that moment. Compare three ways of running a period with a rush:

python -m town.compare cadl/robot_town.cadl --rush 1200 2400 25 --mode DISTRIBUTED
python -m town.compare cadl/robot_town.cadl --rush 1200 2400 25 --mode CENTRALIZED
python -m town.compare cadl/robot_town.cadl --rush 1200 2400 25 --mode adaptive
OperationOn-timeLateBlocked timeTime centralized
fixed distributed202.88.02141 s0%
fixed centralized206.24.6843 s100%
switching by the CADL conditions209.21.61116 s60%

The mode switches three times in the period: to centralized after 5 minutes, to distributed for the rush, and back to centralized after it. Now make the switch take effect without waiting for the road to clear:

python -m town.period --rush 1200 2400 25 --no-quiescence

The last line reports about 1.4 safety-invariant breaches per period (安全条件の違反 1.4 回).

The two switching thresholds are marked safety: true in the constitution. With town.steward in --mode adaptive (the default), any proposal that changes them and improves the KPI is always brought to the human as safety.


10. Run with a real LLM (optional, 15 min)​

The records in §4–§8 come from real runs with Claude Opus 5.5 on 2026-11-16. With your own API key the LLM answers afresh each time, so you will get proposals and recommendations different from the records.

export ANTHROPIC_API_KEY=... # type your own key; never paste it into a chat or a file
python -m town.steward --lang en --mode DISTRIBUTED --rounds 6 --name live-d-en
python -m town.steward --mode DISTRIBUTED --rounds 6 --name live-d # Japanese; KPI in kpi.md

Each period sends up to three requests: rewriting the KPI (when it changed), proposing amendments, and writing a decision request (only when one is needed). The LLM's outputs are saved in runs/<name>/llm/. Without a terminal and without an answer, the loop writes the decision request to p<k>-request.md and stops; write your answer to a YAML file and rerun the same command with the same --name, adding --answers <YAML>, and the recorded LLM outputs are reused instead of being requested again.

The six periods of the records used about 60,300 input and 48,700 output tokens in English, and 70,700 and 63,700 in Japanese. In the earlier runs (2026-10-29/30), one request took from 20 seconds to 2 minutes; once, a request had no response after 11 minutes and was retried.

The record in Japanese​

examples/steward-d/ holds a run of the same KPI written in Japanese (kpi.md).

python -m town.steward --mode DISTRIBUTED --rounds 6 --name kpi-d \
--llm replay --replay-dir examples/steward-d --answers examples/steward-d/answers.yaml
PeriodReasons for the requestLLM recommendsHuman's choiceResult
1regulation (floor 0.6 → 0.65)R1sanction_lambda 0.5 → 1, the proposal that could be adopted automaticallyv2
2agreement (block_k 3 → 2 vetoed by B and C; 2/5 seeds)keepkeepv2
3agreement (block_k 3 → 2 and hold_limit 60s → 50s, both vetoed by B and C; 1/5 seeds each); LLMkeepkeepv2
4uncertain (hold_limit 60s → 50s, 2/5 seeds)keepkeepv2
5agreement and KPI conflict (sanction_lambda 1 → 0.5); regulation (quorum 2 → 3)keepkeepv2
6LLM (hold_limit 60s → 70s narrows the spread by 0.4; 2/5 seeds)keepkeepv2
  • In period 1 the Japanese LLM proposed doubling the sanction weight, which the English LLM did not. It removed every late delivery on all five seeds and passed the vote with A and B. Only R1, a regulation, needed the human; the LLM recommended R1 and wrote that the sanction change "can be handled separately from R1". A period adopts one option, so choosing R1 would have dropped the sanction change; the recorded answer adopts the sanction change.
  • After that there was little left to improve: late deliveries stayed between 0 and 4.4, and every later proposal improved on at most 2 of 5 seeds (3 of 5 for one in period 5). The LLM recommended keeping the rules in every period, citing the human's earlier reasons, and the human kept them.
  • In period 5, under the fairness KPI, the LLM proposed undoing the heavier sanctions. The spread narrows on all five seeds, but C goes back to waiting at the intersection: late deliveries rise from 0 to 7.8 and the lowest on-time rate falls to 0.83.
  • In period 6 the LLM wrote that "the constitution's objective is on_time_deliveries (minimum improvement 1.0)", so the proposal would not count as an improvement. That is wrong: town.steward judges improvement by the KPI's main goal, here utility_gap, and does not use the constitution's objective and min_improvement (only town.council does). The proposal call is given the whole constitution, and the LLM read its objective as the gate.
  • Under the proxy KPI (§8), the Japanese LLM proposed both the heavier sanction weight and the loosening, and flagged the loosening.

What the real runs showed​

  • Recommendations vary. The English and Japanese runs had the same KPI, yet the LLMs proposed different amendments in period 1, and the runs parted there. In the earlier records of this course (2026-11-15), the English D1 recommendation was R1 rather than proposal 2; in the records of 2026-10-29/30, the Japanese D1 recommendation changed when the same request was sent again. The value of a decision request lies less in its recommendation than in laying out the question, who gains and loses, and what is unknown.
  • The LLM's unknowns point at gaps in the tool. The first version of the table did not show the values of the watched goals; the LLM wrote that it did not know "on_time_rate_gap が具体的に どれだけ悪化したか(表には最低の定時率しか載っていません)" ("how much on_time_rate_gap worsened; the table only shows the lowest on-time rate"), so the table gained a column of watched goals and conditions. Not knowing the quorum (English period 2, §6) and taking the constitution's objective for the gate (Japanese period 6) are not fixed yet (§11).
  • Past reasons shape later recommendations. In period 2 the LLM cited D1's promise to re-measure, in period 5 the floor the human set in K5, and in period 6 D5.
  • The simulation shapes the proposals and the decisions. In the records of 2026-11-15, the companies did not respond to the sanction weight, so switching off sanctions changed no delivery number and only narrowed the spread, and the English LLM recommended it. Now that lighter sanctions mean longer waits at the intersection, neither LLM recommends them.

A real LLM record: the LLM as a council​

examples/llm-distributed/ holds two periods of proposals written by Claude Opus 5.5 on 2026-11-16, with the LLM acting as the council of town.council (--proposer llm): rules only, no KPI, no decision requests, adoption settled by the gates.

python -m town.council --proposer replay --replay-dir examples/llm-distributed --mode DISTRIBUTED --rounds 2
PeriodLLM proposalCheckConstitutionImprovementVotes (A B C)
1 (normal)block_k 3 → 2○○○○ × ×
hold_limit 60s → 50s○○×○ × ×
2 (normal + rush)hold_limit 60s → 50s○○×○ ○ ×
block_k 3 → 2○○○○ × ×
  • All four were well-formed amendments, but nothing was adopted in either period.
  • The LLM did not propose sanction_lambda 0.5 → 1, which the rule-based council adopts in period
    1. This request does not include the companies' policies (town/policies.py), so the LLM does not know that the sanction weight changes how C behaves.
  • In period 1 the LLM predicted that B "will vote yes, since its 849 blocked seconds fall"; B voted no. It predicted B's vote from the town's interest rather than B's.
  • One call used about 5,300 input and 7,000 output tokens.

The record carries a digest of the request (request_digest). On replay, the request is rebuilt from the current rules, constitution, amendment history and prompt; if it differs from the recorded one, the replay prints a note.


11. Stretch goals (optional)​

GoalHint
Tell the LLM the quorumPass the constitution's vote and each proposal's vote-gate result to request_request() in town/llm_steward.py. Does the period-2 misreading ("a second override") disappear?
Do not show the LLM values the loop does not useRemove objective and min_improvement, which town.steward does not use, from the constitution passed to propose_request() in town/llm_steward.py. Does the Japanese period-6 misreading disappear?
Tell the LLM how the companies behaveDescribe the companies of town/policies.py in the request of town/llm_council.py. Does the LLM council start proposing sanction_lambda 0.5 → 1?
Break the human's assumptionMake C in town/policies.py withdraw from the town (stop taking orders) as its sanctions or utility worsen, and see what happens when D1's assumption ("B and C keep delivering") fails
Add a reason to askAdd a condition to triage() in town/steward.py, such as "ask when any company's utility falls by 10 or more"
Add a metric and use it in a KPIAdd, say, the largest blocked time of any company to METRICS in town/kpi.py, and write a KPI that uses it. Does it answer the period-5 concern that no metric measures each company's change from road use alone?
Give amendments an expiryAdopted amendments are never repealed, even when the environment changes back. Add a rule that an amendment lapses unless re-adopted within N periods
Stop the proxy drift with regulationAdd invariants on blocked time or late deliveries to invariants and regulation in constitution.yaml, and to invariants_ok() in town/council.py
Open the GSN in another toolWrite a script that exports runs/<name>/gsn.yaml to the format of the GSN editor you use

12. Research exercise: speed of amendment and transition disruption (optional, several hours)​

To reach the KPI sooner, it is tempting to change the rules a lot at once. If the people who follow the rules are humans, large sudden changes confuse them, and until they adjust they keep acting on the old rules. Robots read the new rules at once and are not confused — but that does not mean the transition causes no problems. This exercise asks how the trade-off between reaching the KPI quickly and the size and frequency of rule changes depends on whether humans or robots follow the rules.

The hands-on already gives some clues:

  • The constitution's "one parameter by one step per amendment" (step in constitution.yaml) is an upper limit on the speed of change.
  • In the six periods of the rule-based council used in §8 (seasons-d), the sanction weight went from 0.5 to 1 in period 1, then hold_limit went 60 → 70 → 80 s in the busy season, with the default step of 10 s, and nothing changed in the quiet season. This is not overshoot: it follows the demand that scenario.yaml sets for each period. A rule that tracks a changing environment moves even when every step is right, so separating tracking from overshoot is part of the exercise.
  • In §9, in a period with a rush, switching the operating mode without waiting for the road to clear broke the safety invariant about 1.4 times per period, with robots only. When the rules change, orders and waits begun under the old version are still running.

Exercise 12-1: size of amendments and speed to the KPI (robots only)​

Change step in constitution.yaml (for example hold_limit from 10 to 20 or 30, block_k from 1 to 2) and run the rule-based council for six periods. Choose values so that one step from the current value stays within min–max (the council does not generate proposals outside the range). The constitution is read from the repository root, so use a different --name per condition and restore the file afterwards with git checkout constitution.yaml.

The demand changes from period to period (scenario.yaml), so a back-and-forth may be the council following the demand. To see overshoot itself, also run each condition with every period set to the same demand (for example, all six periods busy), and restore the file with git checkout scenario.yaml.

python -m town.council --mode DISTRIBUTED --rounds 6 --name step30
python -m town.compare runs/step30/v*.cadl --mode DISTRIBUTED # compare every version on seeds the loop never used

For each condition, tabulate:

QuantityHow to get it
Periods to reach the KPIFix a target (for example "at most 2 late deliveries in town.compare"). In which period was the first version meeting it adopted?
Cumulative amendmentSum, over the amendments in history.md, the amount moved divided by the original step
OscillationThe number of times a parameter was moved in the opposite direction to its previous amendment
Loss during the transitionThe sum, over the periods before the target is reached, of the shortfall from the target

How do the periods to the target, the oscillation and the transition loss change as step grows?

Exercise 12-2: a company with human drivers​

Add a company driven by humans to town/policies.py. Choose your own model, for example:

  • Right after a rule change, with probability p a driver still acts on the old rule (waits up to the old hold_limit, and so on); p falls period by period (a learning curve).
  • The larger a change, and the more changes in a row, the larger p and the slower its recovery.

Each company's behaviour is decided only by the rules that Town in town/world.py reads with self.rules = read_rules(ir); it does not know the previous version. Let Town take the previous version's IR (or rule values) and "periods since the change", and pass them from evaluate() in town/period.py and from town/council.py.

Then repeat Exercise 12-1 with three compositions: all three companies robots, one company human, all three human. For each composition, plot speed to the KPI against disruption during the transition (transition loss and violations), and see how the best step changes.

Exercise 12-3: how changes take effect​

Transitions matter even with robots only. Add the following ways of putting a version into effect, and compare violations, late deliveries and safety-invariant breaches during the transition:

  • Notice period: an adopted version takes effect N periods later instead of the next period.
  • Phased introduction: a large change takes effect in small steps over several periods.
  • Transitional provisions: orders and waits begun before the change keep the old version's deadlines.

Transitional provisions are also a question for the CADL specification. Up to cadl 0.3.31 CADL could not say which version's deadlines and monitors apply to a contract instance that is already running. From cadl 0.3.32 amendment.transitional of the amendment extension says, per contract, whether running instances keep the old version (keep) or switch to the new one (switch) (Appendix K, K.4, §13). The simulator of the exercise does not read this declaration yet, so try the three ways above by adding them to the code in town/.

Exercise 12-4: let the human and the LLM choose the speed​

With a real LLM (§10), make town.steward choose the speed:

  • Write a KPI such as "increase on-time deliveries quickly, but avoid sudden changes that would confuse the drivers" in kpi.en.md. What measures does the LLM offer, and how does it describe the cost of the transition in its decision requests?
  • Add a condition to triage() in town/steward.py that asks the human about any proposal that moves a parameter by a large amount at once, and combine it with the large step of Exercise 12-1.

Which speed to choose cannot be read off the KPI's numbers; it depends on how the people following the rules take the change. Record the human's decisions and reasons in GSN. Six periods with a real LLM used about 50,000 to 70,000 input tokens and as many output tokens per run in the §10 records (Claude Opus 5.5).

Questions to close​

  • For each composition of followers, what does the Pareto front between speed to the KPI and transition disruption look like?
  • The constitution's step resembles the trust region that limits each policy update in reinforcement learning (TRPO, PPO). What is the same, and what differs?
  • Who should set the speed limit? In which layer should it sit: rule (the council or the LLM may change it), regulation (decided by humans), or contract (the promise to customers)?

13. What the specification cannot express yet​

Building this exercise exposed five gaps in CADL 0.3. They were all there up to cadl 0.3.31; cadl 0.3.32 adds syntax or IR output for each of them, as the items below say, while the exercise itself still keeps its own files and lets the simulator apply the rules:

  • Up to cadl 0.3.31, cadl sim-ir did not emit sos.transitions[*].safety_invariant into the IR (cadl check did check it as an expression), so the exercise reads it from the checked source file. From cadl 0.3.32 the IR carries it as written in transitions[*].safety_invariant (Appendix A, A.7.1).
  • Up to cadl 0.3.31 an operating mode (regime) was only a name, with no syntax to bind parameter values or contracts to it. From cadl 0.3.32 the block regimes: binds, per regime, values of alpha, beta and lambda of a contract and of entries of context.environment, and the contracts that are active or suspended (Appendix A, A.7.1). cadl sim-ir writes the bindings of every regime, cadl verify reports a conflict between two contracts that no regime has in force together as a warning instead of a failure, the generated Python code monitors a contract only in the regimes in which it is in force, and the stochastic model extension takes the bound values as the parameter values of its designs (Appendix H, H.6). CADL can now say that centralized mode has bays (for example environment.bays: 3) and which contracts hold in it; how robots use the bays and the exclusive edges is still decided by the simulator.
  • Up to cadl 0.3.31, which layer (rule or contract) a parameter belongs to, and who decides its amendment, could be written only in constitution.yaml and in # @amend: comments. From cadl 0.3.32 the amendment extension (amendment: 0.1, Appendix K, K.3) declares each amendable parameter in the model: its layer, the actor or role that decides (for example COUNCIL by vote with a quorum of 2, or a human OPERATOR), min, max, step, how many steps one amendment may move it, and at:, the places of the model that hold the value. cadl check rejects an unknown actor, a value outside the range or off the steps, and a place that does not hold the value. The exercise still reads constitution.yaml and the comments.
  • Up to cadl 0.3.31 there was no syntax for when a version of the rules takes effect, or for which version applies to contract instances already running (transitional provisions; §12, Exercise 12-3). From cadl 0.3.32 a version gives sos.version, the version it replaces, the period or time from which it applies, and per contract whether running instances keep the old version until they end or switch to the new one, at once or when they are in given states (Appendix K, K.4). Switching is the case to which on_amendment: of the norms extension applies (Appendix F, F.6); an instance that keeps the old version is not amended. cadl amend-diff OLD NEW lists the parameters and contracts that changed between two versions and reports every change that the declared amendability does not allow. No tool applies versions while the town runs: the simulator still decides that.
  • Up to cadl 0.3.31 a metrics target was a single expression, with no syntax for a KPI that changes over time, for how a KPI is measured, or to link the record of human decisions (GSN) to a version of the rules. From cadl 0.3.32 a metric may give schedule: (a target from a period, a time or a version on), measure: (for example one period, five seeds, the mean), and a version lists the decision records it rests on (decisions.md#d1, gsn.yaml#S2), which parameters, targets and measurements name (Appendix K, K.5). These are declarations: the KPI itself, in natural language, and the records stay in kpi.md, decisions.md and gsn.yaml.

Up to cadl 0.3.11, an unreadable value such as deadline: soon passed cadl check and became null in the IR; from 0.3.12 it is an error. The exercise also rejects rules whose IR lacks a value the town needs, so the check gate works even with an older compiler.


Summary​

  • The KPI is written by a human in natural language and may change over time. The LLM turns it into a measure, but when it can be read several ways or relies on a proxy, the human chooses.
  • The LLM proposes amendments to rules, regulation and contracts toward the KPI. Matters needing stakeholders' agreement, the promise to customers, regulation, safety, conflicts within the KPI and uncertain improvements it does not decide: it hands them to the human with the measurements, who gains and who loses, and what it does not know.
  • The human's decisions and reasons are kept, in prose and in GSN, alongside the versions, and change the LLM's next proposals and requests.
  • What finally backs an adoption is the gates — check, constitution, measured improvement and the parties' agreement — together with the reasons the human left.

References​