# Control Seat / CODESYS FR202 reproduction kit This kit reproduces the public same-device IEC execution comparison described in **Building a Faster Virtual PLC**. It contains no credentials, customer data, device addresses, or Control Seat compiler source. ## What this kit proves - The ten CODESYS workloads can be regenerated from the exact public script. - Each case uses the same ordered workload definition and input seed recorded in `workload-manifest.json`. - A new set of 256 on-device samples per workload can be summarized with the included offline tool and compared with the published receipt. It does **not** make the CODESYS runtime or Control Seat compiler open source, replace a CODESYS license, or turn the result into an independent product certification. It measures compiled IEC program execution, not task-release jitter, I/O latency, motion, visualization, or functional safety. ## Included files - `codesys-benchmark-project.py` — creates and builds the complete CODESYS benchmark project through the official CODESYS ScriptEngine. - `workload-bodies/*.st` — the canonical shared Structured Text statement body for each of the ten workloads, exactly as both runtimes executed it. - `workload-manifest.json` — ordered case list and SHA-256 receipts for the canonical workloads. - `results-codesys-fr202.json` — published environment, methodology, medians, p99 values, package hashes, and claim boundary. - `summarize-codesys-samples.py` — validates and summarizes exported samples. - `SHA256SUMS` — integrity receipt for every file in the kit. ### Verifying that both runtimes ran the same program `shared_body_sha256` is the SHA-256 of the workload's statement body — the statements joined with `\n` and terminated by one trailing newline. That is exactly the content of the matching file in `workload-bodies/`, so you can recompute every entry yourself: ```sh shasum -a 256 workload-bodies/timers-50.st ``` Compare each digest with `shared_body_sha256` for the same case in `workload-manifest.json`. The Structured Text that CODESYS compiled is embedded in `codesys-benchmark-project.py`; the only permitted difference from the shared body is the documented dialect normalization described below. This is the check that makes the comparison fair rather than merely reproducible — without it, the manifest is a set of hashes over text you cannot see. ## Required environment - Windows with CODESYS Development System 3.5 SP22 Patch 3. - Standard library 3.5.22.0. - CODESYS Control for Linux ARM64 SL 4.21.0.0 on an ARM64 target. - For the published comparison: OnLogic FR202, Ubuntu ARM64, Cortex-A72, `ondemand` governor, and a 20 ms CODESYS task. - Python 3 only for the offline summarizer. The project-builder script runs inside CODESYS and uses its bundled ScriptEngine. The CODESYS runtime may require a trial or purchased license. Licensing does not change the workload source, but it can limit how long the runtime executes. ## 1. Verify the kit On macOS or Linux: ```sh shasum -a 256 -c SHA256SUMS ``` On Windows, compare each entry with: ```powershell Get-FileHash .\codesys-benchmark-project.py -Algorithm SHA256 ``` ## 2. Generate the CODESYS project Create `C:\Benchmark\CODESYS`, then run the script using the installed CODESYS executable. Installation paths vary, so replace `` with the real path on the test computer: ```powershell & '' ` --profile='CODESYS V3.5 SP22 Patch 3' ` --runscript='C:\path\to\codesys-benchmark-project.py' ` --noUI ``` This produces `C:\Benchmark\CODESYS\control-seat-matrix.project`, adds the official Standard library, creates all ten function blocks, configures the 20 ms cyclic task, builds the application, and exits. Inspect the emitted `MESSAGE|...` lines and do not continue if the build reports an error. CODESYS documents `--runscript` and `--noUI` here: https://content.helpme-codesys.com/en/CODESYS%20Scripting/_cds_starting_script_via_command_line.html ## 3. Deploy to the ARM64 runtime 1. Open the generated project in the same CODESYS profile. 2. Select the Linux ARM64 SL target in Communication Settings. 3. Log in, download the application, and start it. 4. Confirm `PLC_PRG.done` can be observed online before collecting results. 5. Stop Control Seat services during the CODESYS measurement. Stop CODESYS during the Control Seat measurement. Do not run the engines concurrently. Keep the CPU governor, target temperature, background services, runtime versions, and test order in the run notes. A different target is a new result, not a reproduction of the FR202 number. ## 4. Collect every workload The cases are ordered exactly as they appear in `workload-manifest.json`. For each `selected_case` from 0 through 9: 1. Set `PLC_PRG.selected_case` to the case index. 2. Increment `PLC_PRG.command_epoch` once. 3. Wait for `PLC_PRG.done = TRUE` and `PLC_PRG.sample_count = 256`. 4. Export all 256 values from `PLC_PRG.samples[0..255]`. 5. Append them to one UTF-8 CSV with this header: ```csv workload,sample_ns boolean-100,463 boolean-100,470 ``` Do not copy only the median. The summarizer rejects missing, extra, unknown, negative, or malformed samples. ## 5. Summarize the new run ```sh python3 summarize-codesys-samples.py samples.csv > rerun-summary.json ``` The output contains the sample count, median, p99 (nearest-rank), minimum, and maximum for every workload. Compare that receipt with `results-codesys-fr202.json`; keep both even when the numbers differ. ## Control Seat side The published Control Seat rows were produced by its native AOT benchmark harness using the compiler, flags, warm-up count, iteration count, and builder image recorded in `results-codesys-fr202.json`. Evaluation customers can ask Control Seat for a matching signed ARM64 benchmark build. The public kit does not disclose the proprietary compiler implementation, so this is a repeatable engineering benchmark—not an independent source-code audit. ## Known limitations of this matrix Read these before quoting any row: - **The timer workload does not exercise on-delay timing.** `timers-50` toggles the timer input every scan (`phase := NOT phase`), so every TON instance is restarted before it can reach its `T#100ms` preset. No instance ever sets `Q`. Both runtimes execute only the start and reset paths, so the row compares function-block call overhead — specifically the cost of acquiring time — and not elapsed-time behaviour. - **Control Seat receives its scan timestamp as a parameter.** Its generated scan function takes `now_ns` as an argument, and the benchmark harness supplies a synthetic monotonically increasing value rather than calling the clock. Production reads the monotonic clock once per scan; that single read is therefore outside the measured region. CODESYS pays real time acquisition. The 60× timer figure is best read as the upper bound of the scan-coherent timestamp advantage, not its net value in a shipped system. - **The two engines are sampled differently.** CODESYS reports the median of 256 on-device samples of 1,000 executions each. Control Seat reports the median of five samples of 200,000 executions each. The Control Seat figure is the median of a much smaller sample set, and no Control Seat p99 is published, while `codesys_p99_ns` is. Treat the medians as comparable and the tails as characterized on one side only. - **The row CODESYS wins is also the noisiest.** On `integer-1000`, the CODESYS p99 (32,202 ns) is about 1.9× its median (16,838 ns), so a single run can land on either side of the line. ## What this result covers These measurements support one statement: > On the same FR202, Control Seat's native compiled engine led the official > CODESYS Linux ARM64 runtime on 9 of 10 tested IEC workloads, including both > PI-style control-loop cases. They say nothing about I/O stacks, task scheduling, motion, or safety-certified control. Those are measured separately, and a compute benchmark does not stand in for them.