balkisoues/lf-delay-monitor-policies

LF Delay-Monitor Policies

Trained PPO and SAC expert and delay-distilled learner policies for the HalfCheetah-v5 locomotion task, produced as part of research on runtime monitoring of observation-delay-tolerant RL policies using Lingua Franca (LF).

This repository accompanies the paper:

"Runtime Monitoring of Observation-Delay Tolerant RL Policies in Lingua Franca" (submitted to IEEE Embedded Systems Letters / TCRS 2026)

Code: balkisoues/lf-delay-runtime-monitor

Overview

Policies are trained on HalfCheetah-v5 under two reward conditions and distilled to tolerate varying levels of observation delay using DIDA (Delayed Imitation with Dataset Aggregation). An LF-based runtime monitor enforces a statistical delay-robustness envelope as a hard deadline over the deployed policy.

• Expert policies: trained with full (undelayed) observations, used as the imitation target for distillation.
• Learner policies: distilled via DIDA to remain robust under a fixed observation delay d (in environment steps).

Naming Convention

{algorithm}_{condition}_{role}[_{delay}]

Component	Values	Meaning
algorithm	ppo, sac	RL algorithm used to train the expert
condition	nr, sr	nr = no reward shaping (baseline), sr = RBF-based reward shaping
role	expert, learner	expert = undelayed teacher policy, learner = DIDA-distilled delay-tolerant policy
delay (learners only)	d3, d5, d7, d10, d20	Observation delay (in steps) the learner was distilled to tolerate

Examples:

• ppo_sr_expert.zip → PPO expert, reward-shaped, no delay
• sac_nr_learner_d10/ → SAC learner, no reward shaping, distilled for a 10-step observation delay

Repository Structure

├── ppo_nr_expert.zip              # PPO expert, no shaping
├── ppo_sr_expert.zip              # PPO expert, RBF reward shaping
├── sac_nr_expert.zip              # SAC expert, no shaping
├── sac_sr_expert.zip              # SAC expert, RBF reward shaping
├── ppo_nr_learner_d3/             # PPO learner (nr), delay = 3 steps
├── ppo_nr_learner_d5/
├── ppo_nr_learner_d7/
├── ppo_nr_learner_d10/
├── ppo_nr_learner_d20/
├── ppo_sr_learner_d3/             # PPO learner (sr), delay = 3 steps
├── ppo_sr_learner_d5/
├── ppo_sr_learner_d7/
├── ppo_sr_learner_d10/
├── ppo_sr_learner_d20/
├── sac_nr_learner_d3/
├── sac_nr_learner_d5/
├── sac_nr_learner_d7/
├── sac_nr_learner_d10/
├── sac_nr_learner_d20/
├── sac_sr_learner_d3/
├── sac_sr_learner_d5/
├── sac_sr_learner_d7/
├── sac_sr_learner_d10/
├── sac_sr_learner_d20/
└── LICENSE

Each learner_dX folder contains the distilled policy checkpoint(s) (.pt) for that delay setting.

Training Details

• Environment: HalfCheetah-v5 (MuJoCo, Gymnasium)
• Algorithms: PPO and SAC (Stable-Baselines3)
• Distillation: DIDA (Delayed Imitation with Dataset Aggregation), rolling buffer capped at 10 iterations
• Reward shaping (sr condition): RBF-based shaping on normalized velocity tracking error
• Architecture: 2×256 hidden layers (learners); some ablations use 3×100
• Seeds: multiple seeds per condition (5 for experts, as reflected in the paper's experimental results)

Intended Use

These checkpoints are released for reproducibility of the accompanying paper — reproducing training curves, delay-robustness envelopes, and runtime monitor evaluation results. They are research artifacts, not production-ready controllers.

Loading a Policy

from stable_baselines3 import PPO, SAC

# Example: load a PPO expert
model = PPO.load("ppo_sr_expert.zip")

# Example: load a SAC learner distilled for 10-step delay
# (adjust to your actual checkpoint filename inside the folder)
model = SAC.load("sac_nr_learner_d10/<checkpoint_name>.pt")

License

Released under the MIT License.