The state of Continuous Integration & Deployment for OT Systems
The gap between how OT teams deploy today and how they could deploy is not a technology problem. The technology exists and the evidence is overwhelming. What it takes now, is the decision to start.
A message from our CTO.
Thank you for taking the time to read this article that examines the evidence, maps platform support and lays out a practical path forward.
We would love to hear what you think. Maybe there is a topic you would like to see us tackle next. If so, please get in touch.
Till then, stay curious.
Philip De Keulenaer
Chief Technology Officer at Agidens
Integration and deployment remain the most manual, error-prone and expensive phases of any SCADA or PLC project.
A decade of empirical research in IT shows clearly that automating integration & delivery pipelines means deploying more frequently, failing less often and recovering faster. The OT vendor landscape is beginning to respond, but readiness varies wildly.
OT systems are lagging 20 years behind these best practices, giving rise to two fundamental questions. Can OT shift to more automated integration & delivery pipelines, and crucially, should it?
The manual reality of OT deployments
Ask any SCADA integrator how a change gets from development to production and you’ll notice that the answer is depressingly consistent.
An engineer develops on a local copy of the system, one that hopefully reflects what is actually running in production. Changes are logged by hand, usually in a Word document with screenshots.
Deployment means dialing into an engineering station, working through the change log line by line against the live system and dragging objects across the screen. For a single engineer making a small change, the time spent during & preparing for deployment, can easily exceed the time spent developing by a factor of 5x.
On larger teams, a new problem arises: coordination. Basic concepts like versioning and merging different branches of code, are barely supported. Engineers work on copies of the system and decide in a team meeting who can touch which blocks, files or screens. The system is far from waterproof, so extra manual controls are implemented like backups on a SharePoint site and rigorous change logs that are manually recorded.
The consequences are far reaching. Programming PLC/SCADA systems entails a surprising amount of integration & deployment “administration”, increasing cost and lead time. Deployments are error-prone, expensive, complex and hence less frequent. Requiring operators to wait months for even the smallest quality of life improvements.
Case studies
1. Small changes to SCADA system
An engineer opens Studio 5000 to make a small change for a food production customer.
The development takes about an hour: a new screen, a handful of logic tweaks … But the development takes about an hour.
Every change has to be documented as she goes: a Word file listing every block she touched, every screen she modified, annotated with screenshots. The documentation takes longer than the work itself.
She tests on her local copy of the system, which is based on a 3-day old backup from production. When the deployment window arrives, she dials into the customer’s engineering station, works through the change log against the live system item by item and drags screens and objects across one at a time, ticking each off in the document. She is working on the live system, while production is running.
Total time: two days. Development: about an hour. Effective deployment? About an hour. The rest is documentation and careful checking.
2. Traceability of changes
Eight engineers work on a large Wonderware SCADA project, checking out sections of code, making changes and checking them back in against a central version.
On check-in, the system compiles to catch syntax errors. There are no other automated checks and no practical way for a lead engineer to review changes before they are merged.
On Friday, the team deploys a batch of changes to production. On Monday, operations reports that a feature working fine last week is now behaving differently. The cause: production had been patched directly weeks earlier, a small fix that never made it back into the development environment. The deployment silently overwrote it.
There is no way to compare the development version against what is actually running in production. The lead engineer pieces together what happened from SharePoint backups, the change log document and the memories of the people involved.
The most basic question in engineering has no reliable answer: what changed?
History repeats itself
The reality of OT development today is similar to that of IT, perhaps 2 decades ago. In the meantime, IT development has taken massive steps forward in tools to support the development, integration & testing of new software.
This resulted in the principle of Continuous Integration & Continuous Deployment (CI/CD), pipelines built on a rich ecosystem including Git (version control), Docker (containerized deployment), Jenkins (pipelines), ELK (monitoring) and many more, often managed by a dedicated “DevOps engineer”.
Fundamental research shows two surprising findings for companies who excel in CI/CD:
- Companies who ship features fastest, have less issues and faster recovery than those who ship slowly.
- Extenisive approval processes do not reduce failure rates, but increase lead time, reduce deployment frequency and reduce time to restore.
It’s no surprise therefore that >80% of software companies have CI/CD practice in place, and the market for DevOps engineers is growing >20% per year.
>80% of software companies have CI/CD practices in place, and the market for DevOps engineers is growing >20% per year.
Google backed DORA[1] (DevOps Research & Assessment) analyses the impact of DevOps on software team’s productivity and the evidence is striking.
- 182 x more deployments per year than low performers
- 127 x faster lead time from commit to production
- 8 x lower change failure rate
- 2,293 x faster recovery from failed deployment
This is counterintuitive for anyone trained in OT’s cautious culture: speed and stability are not a trade-off. The fastest teams are also the most stable. Teams that deploy more frequently fail less often and recover faster.
In their landmark book, Accelerate[2], Forsgren, Humble and Kim found that formal external approval bodies, change advisory boards and manual sign-off gates were negatively correlated with lead time, deployment frequency and restore time, with no positive correlation to change failure rate.
Manual approvals slow things down without making them safer. This does not mean approvals are unnecessary in safety-critical environments. It means they need to be embedded in the pipeline, not layered on top as a separate bureaucratic step.
[1] dora.dev/research/publications
[2] Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations. IT Revolution Press. — Chapter 4, in the section on change approval processes
A possible solution blueprint
It’s clear that the enterprise IT industry has laid down a blueprint to deliver new software rapidly, cheaply & safely with CI/CD pipelines at the heart. The core principles of CI/CD however, can easily extend to OT.
Continuous Integration means that every change, whether a PLC logic edit, an HMI screen modification, an alarm threshold adjustment or a gateway configuration update is committed to a shared version control repository.
On commit, automated checks run: does the project compile? Do tag references resolve? Are alarm limits consistent? Do automated tests pass? Errors are caught immediately, and lead engineers can easily review and comment on what their teams are producing.
Continuous Deployment means that the path from development to production is automated and auditable. Configuration differences between environments are managed as code, not as manual steps. Deployment scripts automatically push the specified software release onto the target infrastructure, with full logging of actions and potential issues.
Yet reality is still a far cry from a real CI/CD process. There are two fundamental questions stopping adoption today: can you, and should you?
Is it technically possible?
| Ignition 8.3 | TIA Portal | WinCC Unified | Win CC OA | PCS7 | Studio 5000 | TwinCAT 3 | CODESYS | AVEVA Sys. Plat. | |
| Version control (native text format) | ⚠️ | ⚠️ | ❌ | ✅ | ❌ | 🔧 | ✅ | ⚠️ | ❌ |
| Automated build & validation | ⚠️ | 🔧 | ❌ | 🔧 | ❌ | ⚠️ | ⚠️ | ❌ | ❌ |
| Test automation & simulation | ⚠️ | ⚠️ | ❌ | 🔧 | ❌ | ⚠️ | ⚠️ | ❌ | ❌ |
| Deployment automation (API) | ✅ | 🔧 | ❌ | 🔧 | ❌ | ✅ | ⚠️ | ❌ | ❌ |
| Config as code / env. management | ✅ | ❌ | ❌ | 🔧 | ❌ | ❌ | ❌ | ❌ | ❌ |
| Container & orchestration | ✅ | ❌ | ❌ | 🔧 | ❌ | ❌ | ❌ | ❌ | ❌ |
Can you?
Fortunately, vendors have taken notice. Every major platform is moving toward Git support, text-based formats and API-driven deployment. For example, Inductive Automation redesigned Ignition 8.3 around file-based configuration and REST APIs. Siemens introduced Git-compatible exports in TIA Portal V21. Rockwell published a CI/CD reference architecture.
Despite the limitations, there are also opportunities. As vendors introduce new options it’s still up to the engineers to leverage them. Even binary backups can be stored on git and textual exports where supported can be checked for tag consistency with a simple python script. The advent of Coding Agents also creates new opportunities to review complex OT codebases. Web-based SCADA’s can be automatically tested with frameworks like Playwright or Cypress, etc.
OT vendors are enabling CI/CD. Engineers now need to turn those tools into workflows.
Should you?
Continuity is the single biggest priority for any OT system, and its key differentiating quality to most of its IT brothers & sisters. Only in OT does time directly equal money. 30 minutes of non-production is at least 2% less revenue for the day.
On top of this OT platforms often control highly sensitive physical processes which, if not managed well, can lead to explosions, pollution or public health disasters. Engineers and customers alike have hence adopted the instinct to be cautious when it comes to releasing new changes into production.
Yet this is contrary to the DORA evidence reviewed earlier. Fast, frequent change leads to less issues. Does this lesson translate cleanly to OT? We honestly do not know, as the state of practice isn’t there yet to prove it one way or another.
One way to start collecting the evidence is to leverage CI/CD not for its fast deployment, but for its auditable and repeatable process. For example, CI/CD turns manual procedures into executable code, reducing risk of human error. Audit logs note exactly what was deployed, to where & when, eliminating the need for manually authored installation records. Automated code checks can reduce error rates during FAT/SAT procedures, etc.
In OT, CI/CD is not about deploying faster first. It is about making every change safer repeatable and fully traceable.
Yes we can, and yes we should
We are confident that CI/CD processes have a place in OT development. The industry is clearly shifting there, but like with all change in OT, it is slow and cumbersome.
We believe we’re in the midst of an acceleration. AI driven development has lowered the barrier to create new kinds of tools that interface with the legacy platforms for code review, simulation, testing…
The space is moving fast, those who want to stay relevant should start now, and focus on safety first.
Safety first
CI/CD processes in IT tend to be safer, faster and cheaper ways of deploying software. OT should focus on the safer part first. Tracking every change, catching bugs early, creating auditable evidence in line with industry standards.
The value of faster deployment and cheaper cost of change, will then follow naturally.
Start now
Inevitably the possibilities are constrained by what OT platforms allow. Yet (small) steps can be made with all of them.
Whether it’s managing your Ignition project in a git repository, or building some code quality scripts on top of Rockwell L5X. These nudges will instill a CI/CD practice and stimulate a search for more, much needed innovation.
Case study: CI/CD for a regulated medtech environment
Ignition 8.3 stores all gateway configuration as files on disk rather than in an internal database. That means everything can go into Git: database connections, tags, screens, alarm pipelines, gateway settings… Engineers can see exactly what changed, who changed it and why, the same way any software team would.
A team building a 21 CFR Part 11-compliant MedTech application used this to set up a full CI/CD workflow. One Git repository holds everything across all environments. Engineers work on feature branches, submit pull requests for peer review and merge into a shared develop branch.
From there, release candidates are built, automatically tested, deployed to a QA environment for validation and promoted to production through a single pipeline. Ignition’s Deployment Modes make this practical: one repository serves every environment. The base configuration stays the same and environment-specific differences like database endpoints or simulated devices are handled automatically depending on whether the gateway is running in dev, QA or production.
The result: deployments are repeatable and fast. Rolling back means reverting a Git commit. Every change has a full audit trail from feature request to production, exactly what a 21 CFR Part 11 auditor needs to see. The team spends its time engineering, not documenting and manually migrating.
Continue the conversation
Let’s build pipelines and practices that turn SCADA deployment from a liability into a strength.
