Adaptive Posture Regulation with RL

A reinforcement-learning project for EPFL's EE-568, tackling static-posture harm from prolonged tablet use: a small on-screen window is moved to prompt the user to shift posture at a healthy frequency. The control problem is cast as a stochastic Markov decision process on a 6x6 grid (a 36-element state space) where the agent chooses how to move the visual stimulus and is rewarded when the user actually changes posture.

Realism comes from the simulated users. Each artificial user has its own posture-change transition probabilities (drawn in 0.3-0.7, tuned so a random policy nets zero expected reward) and its own fatigue parameter λ, with responsiveness decaying as F(t) = 1 − e^(−λt). This non-stationary, noisy environment is exactly where naive Q-learning overestimates values.

Two noise-robust tabular algorithms were implemented and compared. Double Q-Learning keeps two Q-tables QA and QB and, on each step, updates one using the other's estimate of the greedy action (QA(s,a) += α[r + γ·QB(s', argmax QA(s',·)) − QA(s,a)]), decoupling action selection from evaluation to remove the maximization bias. Expected SARSA replaces the sampled next action with the policy-weighted expectation of future value, giving smoother, lower-variance updates. Both learned effective policies; adding fatigue made cumulative reward peak then gently decline without the model collapsing.