Spaces:
Running
Running
| spec_version: 1 | |
| name: netweaver_sre | |
| type: space | |
| runtime: fastapi | |
| app: server.app:app | |
| port: 7860 | |
| tasks: | |
| # ββ EASY ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| - id: netweaver_sre_t01 | |
| difficulty: easy | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t01 | |
| description: > | |
| Node Offline Triage. Read hardware_logs to find the offline node. | |
| Remediate with DRAIN_TRAFFIC on the identified node. | |
| Scored on: diagnosis (40%) + resolution (40%) + best practice (20%). | |
| - id: netweaver_sre_t02 | |
| difficulty: easy | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t02 | |
| description: > | |
| Stale DNS Cache. Read hardware_logs to identify the node. | |
| Issue CLEAR_DNS_CACHE on the affected node. | |
| Scored on: diagnosis (40%) + resolution (40%) + best practice (20%). | |
| - id: netweaver_sre_t03 | |
| difficulty: easy | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t03 | |
| description: > | |
| OOM Crash. Read hardware_logs for the crashed service name. | |
| Issue RESTART_SERVICE on that service. | |
| Scored on: diagnosis (40%) + resolution (40%) + best practice (20%). | |
| - id: netweaver_sre_t04 | |
| difficulty: easy | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t04 | |
| description: > | |
| TLS Certificate Expiry. Read hardware_logs for the affected node. | |
| Issue RENEW_CERTIFICATE on that node. | |
| Scored on: diagnosis (40%) + resolution (40%) + best practice (20%). | |
| - id: netweaver_sre_t05 | |
| difficulty: easy | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t05 | |
| description: > | |
| Disk Space Full. Read hardware_logs for the full node. | |
| Issue CLEAR_TEMP_FILES on that node. | |
| Scored on: diagnosis (40%) + resolution (40%) + best practice (20%). | |
| - id: netweaver_sre_t06 | |
| difficulty: easy | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t06 | |
| description: > | |
| Unhealthy Pod in CrashLoopBackOff. Read hardware_logs for the pod name. | |
| Issue RESTART_POD on that pod. | |
| Scored on: diagnosis (40%) + resolution (40%) + best practice (20%). | |
| - id: netweaver_sre_t07 | |
| difficulty: easy | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t07 | |
| description: > | |
| Zombie Process filling PID table. Read hardware_logs for the node. | |
| Issue KILL_ZOMBIE_PROCESS on that node. | |
| Scored on: diagnosis (40%) + resolution (40%) + best practice (20%). | |
| # ββ MEDIUM ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| - id: netweaver_sre_t08 | |
| difficulty: medium | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t08 | |
| description: > | |
| PFC Buffer Congestion. Read queue_depths for the switch near 99.9. | |
| Issue TUNE_PFC_THRESHOLD on that switch with a numeric value (1000-9000). | |
| Grader checks correct target AND value in valid range. | |
| Efficiency penalty applied for steps beyond ideal (3). | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| - id: netweaver_sre_t09 | |
| difficulty: medium | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t09 | |
| description: > | |
| Power Throttling. Read hardware_logs for node and recommended wattage. | |
| Issue ADJUST_POWER_CAP on that node with the watt value. | |
| Efficiency penalty applied for steps beyond ideal (3). | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| - id: netweaver_sre_t10 | |
| difficulty: medium | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t10 | |
| description: > | |
| BGP Route Flapping. Read hardware_logs for router and AS number. | |
| Issue MITIGATE_ROUTE_FLAP on that router with the AS number as value. | |
| Efficiency penalty applied for steps beyond ideal (3). | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| - id: netweaver_sre_t11 | |
| difficulty: medium | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t11 | |
| description: > | |
| Jumbo Frame Packet Drop (MTU mismatch). Read queue_depths/logs for switch. | |
| Issue INCREASE_MTU on that switch with value exactly 9000. | |
| Efficiency penalty applied for steps beyond ideal (3). | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| - id: netweaver_sre_t12 | |
| difficulty: medium | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t12 | |
| description: > | |
| DDoS API flood. Read hardware_logs for the gateway name. | |
| Issue SET_RATE_LIMIT on the gateway with a numeric req/s value. | |
| Efficiency penalty applied for steps beyond ideal (3). | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| - id: netweaver_sre_t13 | |
| difficulty: medium | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t13 | |
| description: > | |
| DB Connection Pool Exhaustion. Read hardware_logs for DB target. | |
| Issue SCALE_CONN_POOL on that DB with the new pool size as value. | |
| Efficiency penalty applied for steps beyond ideal (3). | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| - id: netweaver_sre_t14 | |
| difficulty: medium | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t14 | |
| description: > | |
| CPU Context Switch Storm. Read hardware_logs for node and thread count. | |
| Issue PIN_CPU_THREADS on that node with thread count as value. | |
| Efficiency penalty applied for steps beyond ideal (3). | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| # ββ HARD ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| - id: netweaver_sre_t15 | |
| difficulty: hard | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t15 | |
| description: > | |
| Silent NaN Contagion. Read gradient_variances for ranks with value -1.0. | |
| Multi-step: (1) RUN_MINI_ITERATION on affected cluster to isolate, | |
| then (2) DRAIN_TRAFFIC on same cluster to remove from pool. | |
| Both commands required for full resolution score. | |
| Efficiency penalty applied for steps beyond ideal (4). | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| - id: netweaver_sre_t16 | |
| difficulty: hard | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t16 | |
| description: > | |
| Broadcast Storm. Read queue_depths for the switch at 99.9. | |
| Issue ISOLATE_BROADCAST_STORM on that exact switch. | |
| Efficiency penalty applied for steps beyond ideal (3). | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| - id: netweaver_sre_t17 | |
| difficulty: hard | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t17 | |
| description: > | |
| GPU Memory Leak. Read gpu_memory_usage for the cluster index near 1.0. | |
| Issue RESTART_GPU_DAEMON on cluster_<that index>. | |
| Efficiency penalty applied for steps beyond ideal (3). | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| - id: netweaver_sre_t18 | |
| difficulty: hard | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t18 | |
| description: > | |
| Full Cluster Deadlock. All arrays frozen at 0.0 β distributed deadlock. | |
| Issue ISSUE_GLOBAL_ROLLBACK with the exact cluster target from logs. | |
| Efficiency penalty applied for steps beyond ideal (3). | |
| Scored on: diagnosis (40%) + resolution (40%) + best practice (20%). | |
| - id: netweaver_sre_t19 | |
| difficulty: hard | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t19 | |
| description: > | |
| Network Partition. Read queue_depths for the split (0.01 vs 99.9). | |
| Issue REBOOT_LEAF_SWITCHES on the pod with the maxed entry. | |
| Efficiency penalty applied for steps beyond ideal (4). | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| - id: netweaver_sre_t20 | |
| difficulty: hard | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t20 | |
| description: > | |
| Corrupt DB Block. Read logs for the randomized DB cluster target. | |
| Issue PURGE_CORRUPT_BLOCK on that exact DB target. | |
| Efficiency penalty applied for steps beyond ideal (4). | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| - id: netweaver_sre_t21 | |
| difficulty: hard | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t21 | |
| description: > | |
| Cascading Failure Chain. A power fault triggers a CPU context switch storm | |
| which overflows the connection pool. Requires 3 sequential remediations: | |
| (1) ADJUST_POWER_CAP on the throttled node, | |
| (2) PIN_CPU_THREADS on the same node, | |
| (3) SCALE_CONN_POOL on the database. | |
| Order matters. Partial credit for each completed step. | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). | |
| - id: netweaver_sre_t22 | |
| difficulty: hard | |
| grader: | |
| type: deterministic | |
| endpoint: /grader | |
| task_level: t22 | |
| description: > | |
| Silent Gradient Poisoning with Network Amplification. | |
| NaN is corrupting gradient sync AND a broadcast storm is amplifying the corrupted | |
| gradients across the fabric. Requires: | |
| (1) RUN_MINI_ITERATION to isolate NaN source cluster, | |
| (2) ISOLATE_BROADCAST_STORM on the storm switch, | |
| (3) DRAIN_TRAFFIC on the poisoned cluster. | |
| All 3 commands required in this order for full resolution. | |
| Scored on: diagnosis (40%) + resolution+efficiency (40%) + best practice (20%). |