How to use it
- 1. Watch it deteriorateA machine walks states 0 (new) → 4 (failed). Each period it may degrade.
- 2. Set the economicsA planned repair is cheap; an emergency repair after failure is brutal. That gap drives everything.
- 3. Run value iterationStep it and watch V(s) converge and the policy settle into a threshold: repair at or above state s*.
- 4. Trust but verifyThe tool brute-forces all 32 deterministic policies and confirms value iteration really found the best one.
Maintenance MDP
Dynamics
A worn machine can fail without warning: the hazard climbs with wear (zero at state 0, full at state 3). That is what makes running it a gamble - and what gives the emergency price any bite.
Costs
Running costs rise with wear: $0, $5, $15, $40 in states 0–3.
Value iteration
The Markov chain under the current policy. Red states = repair; navy = keep operating. Arrow thickness = transition probability; the moving dot is a simulated machine.
V(s) - expected discounted cost from each state. Value iteration drives these to a fixed point.
Steady-state distribution: where the machine actually spends its life under this policy.
Try these experiments
- Set hazard to 0: the machine can never fail without warning, so the emergency price stops mattering and the policy ignores that slider completely. Raise the hazard and watch it come alive.
- At hazard ≈ 0.35, push emergency repair from $20 to $600: the threshold marches down. Cheap failure → run it into the ground; dear failure → replace early. Preventive maintenance, derived rather than assumed.
- Keep emergency repair at $200 and push planned repair from $50 up to $220 or more: the policy walks from “repair at s≥1” ($50) to “repair at s≥3” ($200) and flips to “repair at s≥4” at $220, so prevention turns irrational and it collapses to run-to-failure. (It still repairs at s≥3 up to about $215, because repairing early also dodges the chance of a sudden failure.)
- Drop γ to 0.70: a myopic manager tolerates more wear, because tomorrow’s breakdown barely counts today.
About this tool
The concept
A Markov Decision Process is a Markov chain you can steer: states, actions, transition probabilities and costs. Solve it and you get a policy - what to do in every state, not just once.
How it works
Bellman: V(s) = mina [ c(s,a) + γΣP(s′|s,a)V(s′) ]. Sweeping that update is value iteration; it is a contraction, so it always converges to the unique optimum.
Winning strategy
This is the machinery under the RL Inventory Agent - but here the model is known, so no learning is needed. When you know P and c, solve; when you don’t, learn.