teacher_agent_dev/ANALYSIS_AND_FIXES.md

Analysis: Why Accuracy Drops and How to Fix

Issue 1: Accuracy Drops at End ❌

Root Causes Found:

1. Evaluation uses NEW tasks each time (line 171-175 in compare_strategies.py)

   • general_accuracy = student.evaluate([generator.generate_task(...) for ...])
   • Creates new tasks every iteration → variance and inconsistency
   • Should use FIXED eval set

2. Forgetting rate too aggressive for 500 iterations

   • Forgetting rate: 0.05
   • After 500 iterations (500 time units): retention = exp(-0.05 * 500) ≈ 0.0
   • All skills forgotten by the end!
   • Retention drops to near-zero after ~50-100 time units

3. Evaluation timing confusion

   • Currently: Learn → Evaluate → Advance time
   • Should be clearer about when evaluation happens relative to forgetting

Issue 2: Accuracy Calculation Method

Current Method:

• Uses student.evaluate(eval_tasks) which:
  • Calls answer() for each task (stochastic, uses randomness)
  • Accounts for forgetting via _get_effective_skill()
  • Returns fraction of correct answers

Problems:

1. Stochastic variance: Random sampling introduces noise
2. Eval tasks regenerated: Different tasks each time = inconsistent
3. Small eval set: Only 10-15 tasks = high variance

Better Methods:

1. Use FIXED eval set generated once at start
2. Use expected accuracy instead of sampled (less variance)
   • Expected acc = mean(prob_correct) over all tasks
3. Larger eval set (50-100 tasks) for stability
4. Separate eval timing: Evaluate BEFORE time advance

Issue 3: Mock vs Real Components

Current Mock Components:

Mock Student:

• ✅ Captures learning and forgetting
• ✅ Per-topic skill tracking
• ✅ Realistic Ebbinghaus curve
• ❌ Simplified learning model (linear skill increase)
• ❌ Stochastic but not as complex as real PPO

Mock Task Generator:

• ✅ Simple template-based tasks
• ✅ Multiple topics and difficulties
• ❌ Fixed templates (not procedural)
• ❌ Limited diversity

Real Components (in MentorFlow):

• Student: Full PPO agent with neural network
• Task Generator: Procedural generation with 15 task families

Will Real Components Be Better?

YES, likely:

1. Real PPO student can learn more complex patterns
2. Procedural task generator provides more diverse tasks
3. Better generalization to unseen tasks
4. More realistic learning curves

BUT:

• Real components are slower to train
• Harder to debug and verify
• Teacher agent algorithm (UCB) should still work

Recommended Fixes

1. Fix evaluation to use FIXED eval sets
2. Reduce forgetting rate or reset time periodically
3. Use expected accuracy for more stable measurements
4. Add evaluation BEFORE time advance option
5. Document evaluation methodology clearly