GSIR 2024 expert Q&A: System integration challenges, trends

Control system integrator trends and insights

• Understand technologies, trends and challenges identified by control system integrators in a Control Engineering roundtable discussion.
• Explore system integrator questions and answers on control system integration project management.
• Learn from system integrators about information technology and operational technology IT/OT integration and cybersecurity especially with industrial system integration projects.

Experts in control system integrators answer questions from Control Engineering and Plant Engineering about system integration technologies, trends and challenges, system integration project management, information technology (IT) and operational technology (OT) convergence and cybersecurity. Leading trends that system integrators (SIs) identified include adoption of AI-driven automation and smart manufacturing, digital transformation, secure networking and data access, software as a service and user interface improvements.

Answering questions on control system integration topics in a roundtable format are:

• Navid Radnia, CSSGB, PMP, MBA, director of corporate PMO, E Tech Group.
• Steve Orth, vice president of automation, The Eosys Group.
• Joshua Payne, PE – AL, GA, SC, controls and automation division leader, Hargrove Controls & Automation.
• Dan Malyszko, vice president, Malisko Inc.
• Thea Sander, technical marketing lead, Vertech.

Five system integration experts answered Control Engineering and Plant Engineering roundtable questions on system integration topics. Starting upper left: Navid Radnia, CSSGB, PMP, MBA, is director of corporate PMO, E Tech Group; Steve Orth is vice president of automation, The Eosys Group; Joshua Payne, PE – AL, GA, SC, is controls and automation division leader, Hargrove Controls & Automation; Dan Malyszko is vice president, Malisko Inc.; and Thea Sander is technical marketing lead, Vertech.

System integrators address technologies, trends, challenges

Control system integrators answered questions about system integration technologies and trends and how they approach system integration challenges.

What is the biggest challenge a system integrator currently faces, and how are you and your company working to overcome it?

Radnia: The biggest challenge system integrators currently face is managing the complexity and rapid pace of technological advancements while ensuring seamless integration across diverse systems and platforms. At E Tech Group, we address this challenge by adopting a hardware and software agnostic approach, allowing us to tailor solutions that best fit our clients’ needs. We invest in continuous learning and partnerships with industry leaders to stay ahead of technology trends. Additionally, our focus on digital transformation and automation optimization ensures that our clients benefit from the most innovative and efficient solutions available.

Orth: Recruiting quality staff is our biggest challenge. Anyone can grow big, but it is hard to do that while maintaining your core values across every facet of the organization. Being a 100% employee-owned company brings intrinsic and financial incentives that set us apart in the eyes of candidates that help us to recruit the best talent.

Payne: Acquiring the right talent continues to be an issue. We can certainly find entry-level talent, but finding mid-career or higher-level talent with the specific skillsets in the specific industry where we serve continues to be difficult. One of the ways we have created a pipeline for early talent is our co-op program. A lot of our current teammates were involved in a co-op program in college. Now being on the employer’s side, we see the value in it. This advantage to our business allows us to identify talent early in their career and recruit those who could fit into our company including values, skillset, etc. For mid-level or higher-level talent, we leverage the relationships of current teammates as well as utilize our recruiting team.

Sander: Managing (human) labor resources is huge. People raise the question of whether you should build up your team before you go after more work or wait until you have projects ready. And you really have to strike a balance to keep the workload even, your team happy and projects running on schedule. This is compounded by project delays, labor rates and a competitive market, so this is industrywide. At Vertech, we focus on maintaining a sustainable growth rate both for hiring and sales, so one doesn’t outpace the other. We’re also dedicating a significant effort to improving efficiency by developing tools and processes that help us work faster and smarter while generating more value for our clients.

Figure 1: Panel shop technician installs a warning beacon and horn for a petrochemical client. Courtesy: Hargrove Controls & Automation

What do you see as the biggest trend happening for system integrators (looking ahead a couple of years)?

Radnia: The biggest trend for system integrators in the coming years is the increasing adoption of AI-driven automation and smart manufacturing solutions. As industries move toward Industry 4.0, the demand for integrating AI, IoT and data analytics into traditional systems is rapidly growing. At E Tech Group, we are positioning ourselves at the forefront of this trend by developing expertise in AI and machine learning integration, enabling our clients to optimize their operations through predictive maintenance, enhanced process control, and real-time data insights.

Orth: Digital transformation continues to reshape our industry. More of our projects engage customers at a higher level who are looking to enhance their ability to make business decisions quickly with the data that is available through our control systems.

Payne: There seems to be a push in the industry to embrace artificial intelligence, digitalization and Industry 4.0, but you need access to data to be able to do any of those. A lot of people don’t have access to the data in the ways they need. There are still a lot of islands of automation where devices are not interconnected, and the data is not accessible. So, executing the right projects to enable the data collection, installing the right OT networks, and ensuring they are secure will enable you to access the data and interpret it in meaningful ways. Also, cybersecurity continues to be a trend. There is no shortage of threat actors in the industry, so there is a need to analyze systems and find vulnerabilities to reduce the threats.

Sander: The biggest trend will be adopting development standards defined by SaaS (software-as-a-service) companies like Figma, Adobe, Google, Uber and other. This is your agile scrum, CI/CD pipeline, automated testing, separate development and test environments, etc. But within that, the big theme we’re seeing is user experience (UX) research [often integrated into the human-machine interface, HMI]. This means taking the time to interview operators or business managers to understand what they need to be successful and make decisions in their day-to-day. Having that human perspective takes the ROI of your project to the next level. Not only that, but incorporating their feedback throughout the project makes for truly user-friendly systems and user interface (UI). We call it human-focused design.

What challenges do you encounter when adopting new technologies, and how do you overcome them?

Radnia: One of the primary challenges when adopting new technologies is ensuring compatibility with existing systems while minimizing downtime. At E Tech Group, we tackle this by conducting thorough testing and simulations before full-scale implementation. For example, when integrating IoT solutions into a legacy manufacturing system, we first developed a digital twin of the system. This allowed us to identify potential issues and fine-tune the integration process, ensuring a smooth transition with minimal disruption to production.

Orth: The primary challenge when adopting new technologies is getting familiar with the IDE/programming interface and how it works. The concepts and languages are very similar across most industrial control system platforms. Recently, we had a customer request that we provide support for equipment using an unfamiliar control system. We invested in training on that platform and are now setup to provide the necessary support.

Payne: When adopting new technologies, one should consider how the technology can be incorporated with your business objectives. How will this grow your business? What training is involved with adopting this technology? How do you invest enough R&D so it continues to generate revenue? To overcome that, one can recruit personnel with the necessary skillsets and training who can seed your team with new skillsets.

Malyszko: The primary challenge in adopting new technologies is ensuring they solve the right problem and deliver tangible value. When we introduce a new IoT platform to a client, for example, the technology alone isn’t enough — it needs the foundational integration work to model and contextualize the plant’s data. This process transforms raw data into actionable intelligence, enabling the client to optimize operations and reduce costs. The key is avoiding tech for tech’s sake; it’s about aligning innovations with real-world needs and ensuring they provide meaningful, measurable benefits.

How do you keep up with rapid technological advancements and integrate emerging technologies into existing systems?

Radnia: At E Tech Group, we stay ahead of rapid technological advancements by fostering a culture of continuous learning and innovation. Our team regularly engages in professional development, training, and industry certifications to keep our skills sharp. We also maintain strong partnerships with leading technology providers, giving us early access to emerging technologies. When integrating these technologies into existing systems, we use a phased approach, starting with pilot projects to ensure smooth and efficient integration before full-scale deployment.

Orth: It is important to subscribe to webinars and be engaged in all other methods of staying up to date on emerging technologies. We believe it is also key to have partnerships with customers where innovation is welcomed with the understanding that growing pains may occur as you’re integrating emerging tech into the solutions.

Malyszko: Keeping up with rapid technological advancements is all about empowering our people. Technology is only as effective as the engineers who apply it, and in today’s world, where everyone is challenged to do more with less, our success hinges on the curiosity and ingenuity of our controls engineers. Staying up with features and benefits of new software releases as well as exploring options for system extensibility allows us to seamlessly integrate emerging technologies into existing systems. By fostering a culture of continuous learning and innovation, our team stays ahead of the curve, ensuring we can provide our customers with the most efficient and effective solutions.

Sander: We’ve leveraged more use-case testing and edge-case testing in the design phase to build proof of concepts before anything being deployed. For example, most clients want to be able to pull real-time data from every piece of equipment on their plant floor. And it’s easily achieved, especially with tools like Ignition by Inductive Automation. But let’s say one of the PLCs on the floor is 15 years old. You might use most of that PLC’s processing power just on aggregating data. So now your process isn’t working because that PLC can’t perform its primary function. That’s where a proof of concept really helps. You build a small solution, deploy it into a simulated test environment, and then scale it out to see what the impact will be.

Figure 2: Panel shop technician inspects wiring terminations for a petrochemical client. Courtesy: Hargrove Controls & Automation

How do you handle unforeseen issues that arise during the integration process?

Radnia: When unforeseen issues arise during the integration process, E Tech Group employs a proactive and agile approach. We start by conducting a thorough root cause analysis to quickly identify the problem. Our multidisciplinary team collaborates to develop and implement a solution, often involving real-time adjustments to the integration plan. For example, if a system incompatibility is detected, we might reconfigure software or hardware components on the fly to ensure continuity. Continuous communication with the client ensures transparency throughout the process.

Orth: There are always “unforeseen” issues on every project. Our job is to proactively plan to mitigate all of the “foreseen” issues so that we will have bandwidth to calmly address any “unforeseen” issues when they arise.

Payne: Do everything you can to limit, minimize or eliminate any unforeseen issues. Start with interactive planning sessions with clients to vet out the unknowns and build in risk mitigation plans to turn unknowns into knowns, conduct field discovery or create contingency plans to handle those risks. In the event that you can’t minimize that risk to zero, ensure open lines of communication and bring up any issues early so that all parties have the opportunity and time to make the best possible decisions.

How do you ensure compliance with industry standards and regulations during integration projects?

Radnia: At E Tech Group, ensuring compliance with industry standards and regulations is a top priority during integration projects. We begin by conducting a comprehensive review of all relevant regulations, including industry-specific and regional requirements. Our team is well-versed in these standards, and we regularly participate in compliance training. Throughout the project, we implement rigorous quality assurance protocols, including documentation, testing, and audits, to ensure that every aspect of the integration meets or exceeds regulatory requirements.

Payne: When it comes to vendors, having subject matter experts on specific systems and adhering to the manufacturer’s best practices are important. If we are talking about things like NFPA code for boilers, for example, having a team of experts that understand those standards, are proficient in those standards, and even help produce those standards by serving on the standards committee goes a long way to ensuring projects are adhering to standards and best practices.

AI has grown in stature over the last few years. How has it helped system integrators do their work?

Radnia: AI has significantly enhanced the capabilities of system integrators by enabling more efficient and intelligent automation processes. It has improved predictive maintenance, allowing for early detection of potential issues before they escalate, thereby reducing downtime. AI-driven analytics help optimize system performance by providing real-time insights and data-driven decision-making. At E Tech Group, we leverage AI to streamline complex integrations, enhance process control, and deliver smarter, more adaptive solutions that evolve with our clients’ needs.

Malyszko: Engineers are natural knowledge seekers, always eager to deepen their understanding and tackle complex challenges. Generative AI has become a powerful tool in this pursuit, enabling us to arrive at solutions more efficiently. It accelerates the learning curve, providing instant access to a vast array of information and best practices, which helps engineers refine their skills and broaden their knowledge. For system integrators, this means staying ahead of industry trends, innovating faster, and delivering more effective solutions to our clients. AI has truly given an edge to those committed to continuous improvement in our field.

Sander: AI is still so new. We’ve seen only the smallest piece of its full potential. There’s greater potential for predictive maintenance, prescriptive data analytics and machine learning in the future. And currently, it can be leveraged to help write a lot of code quickly or build documentation packages. But it’s not at the point where you can fully trust the outcome. So you still have to do your own due diligence via review and testing.

Figure 3: Vertech helped steel pipe maker American SpiralWeld Pipe implement a manufacturing execution system (MES) at its facility to improve operational efficiency and decision-making. Courtesy: Vertech

How do you think AI and machine learning will impact system integration in the future? 

Radnia: AI and machine learning are poised to revolutionize system integration by enabling more adaptive, self-optimizing systems. In the future, AI-driven automation will facilitate seamless integration of disparate systems, allowing them to communicate and adapt in real time. Machine learning will enable predictive analytics, reducing downtime and improving efficiency by anticipating and addressing issues before they occur. AI and machine learning are set to significantly transform system integration by enabling systems to become more intelligent, adaptive and autonomous. Future integrations will likely involve AI that can self-optimize, anticipate issues, and dynamically adjust operations in real time. Machine learning will enhance predictive maintenance, reducing downtime and operational costs. At E Tech Group, we anticipate that these technologies will drive more autonomous operations, enhancing the scalability and flexibility of integrated systems.

Orth: AI will replace some control schemes such as PID, MVC, etc. The setpoint is often a number used to achieve some other goal. AI will be taught the goal and will drive setpoints to achieve the goal.

Payne: It’s possible that one day in the future a lot of the keyboard work and programming that is done today will be handled by AI. Prompt engineering is becoming popular. Having personnel who can prompt the AI to get the technical results and with the knowledge to check the accuracy of the AI output will be important.

Malyszko: AI and machine learning will transform system integration by allowing controls engineers to shift their focus from repetitive tasks to refining and optimizing the processes they control. With AI, bulk configurations can be executed and verified efficiently, drastically reducing manual effort. Machine learning, in particular, will play a crucial role in optimizing process control by analyzing vast amounts of data in real time, identifying patterns, and making adjustments that improve efficiency and reduce downtime. However, it will still require experts with a deep understanding of control systems and enterprise architecture to orchestrate these advancements, ensuring that increasingly complex systems work together seamlessly and effectively.

What challenges have you encountered when upgrading legacy systems and ensuring they work well with the planned changes?

Radnia: Upgrading legacy systems presents several challenges, including compatibility issues with modern technologies and potential disruptions to existing operations. At E Tech Group, we address these challenges by conducting a thorough assessment of the legacy system’s architecture and functionality. We often employ a phased approach to upgrades, allowing for gradual integration and minimizing operational downtime. For instance, during a recent project, we used middleware to bridge old and new systems, ensuring smooth communication and reducing the risk of system failures.

Orth: Server infrastructure migrations are more challenging. With a programmable logic controller (PLC) or distributed control system (DCS), there are limited and known touch points. With a Microsoft Windows operating system, the customization can be more complex and open (registry keys, static routes, services, etc.). We find that it is easier to install fresh versions of software according to best practices and migrate the data.

Payne: Set realistic expectations of the effort, time, and money involved to effectively complete a successful migration. At times, there is a significant reverse engineering effort to understand what exists and what should be used as a basis for design for a new system. When there is a question on how a piece of equipment worked before, do you have to reengineer the code, or do you already have a narrative built? Being able to reverse engineer into a set of narratives that can be discussed in a more natural language before programming starts helps the project go well.

What advice would you give to someone starting in the field of system integration?

Radnia: For someone starting in the field of system integration, my advice would be to focus on continuous learning and adaptability. The industry evolves rapidly, so staying current with new technologies, tools, and methodologies is crucial. Gain hands-on experience with different systems and industries to build a versatile skill set. Additionally, develop strong problem-solving and communication skills, as these are essential for addressing the complex challenges you’ll encounter. At E Tech Group, we value curiosity and a proactive approach to learning.

Orth: You can’t become an expert in our field by reading books or doing lab exercises. Instead, find a company with experienced mentors who are willing to guide you and ask them lots of questions.

Payne: System integration work for a system integrator can be rewarding. It’s rewarding to design something, to program it, to take it to a site to install it, and to see it do all the things that you told it to do. This can sometimes result in late nights, early mornings, and travel, so you have got to be willing to step up and put in the effort to reap the rewards.

Malyszko: Embrace the significance of your role—industrial systems integration is a driving force in modern society. Every time someone flips a switch, turns on the water, or buys a product, it’s the result of the massive technological advancements and the dedication of people in our field. Understand that your work helps keep the world running smoothly, often in ways people take for granted. Stay curious, be persistent, and take pride in knowing that your contributions will make a tangible impact on people’s lives every day.

Sander: Start from the bottom. Start at the device/PLC level, understand the foundations, and then move up the OT stack from there. The people who are the most successful in this industry, whether in a senior-level or an architect role, often start this way.

Can you provide an example of a challenging project and how your team worked to overcome it?

Radnia: One particularly challenging project involved integrating a new automation system into a legacy manufacturing environment with minimal downtime. The key challenge was ensuring compatibility between outdated equipment and modern technologies. Our team at E Tech Group approached this by first creating a digital twin of the existing system to simulate potential issues. We then employed a phased integration plan, working closely with the client to schedule updates during low-production periods. Effective communication, thorough testing, and our team’s expertise were crucial to the project’s success.

System integrator questions and answers on project management

The following questions and answers address project management, a critical element of any controls and automation system integration project.

What are the key project management challenges in system integration projects?

Radnia: Key project management challenges in system integration include coordinating multiple stakeholders, managing timelines, maintaining quality, and ensuring seamless communication between diverse systems. Aligning the goals of different teams while balancing technical complexities can be difficult. At E Tech Group, we address these challenges by implementing robust project management frameworks that emphasize clear communication, regular status updates, and rigorous quality control. We also use advanced project management tools to track progress, manage resources, and ensure that every phase of the integration meets both project objectives and quality standards.

Orth: The biggest challenge in project management is aligning customer expectations. This requires identifying stakeholders, documenting requirements, and demonstrating functionality to avoid surprises.

Payne: Many facilities that need to complete system integration projects have old systems that have been in place for more than 50 years with P&IDs, loop sheets and motor schematics that may not be not up-to-date. These components are critical for a successful system integration project, especially a migration. With talent leaving the field, mostly due to retirement, and sites running with less staff, there are less people in the plant with the experience and knowledge specific for the operations of that site. Site-specific information is lost as many years of knowledge can’t be transferred to a new employee in a short amount of time.

Sander: Two challenges that apply to almost every project are 1) effectively managing and mitigating a project’s technical risks and 2) truly understanding the scope of the project. These are intertwined. The key to effective and profitable project execution is determining the quality level the client is looking for and engineering a solution that fits their needs. If you can nail down these two aspects, you’ll be less likely to need a ton of engineering rework.

Figure 4: SB Energy engaged Vertech to provide a one-platform SCADA system to remotely oversee six solar sites. Courtesy: Vertech

How do you manage timelines and resources effectively to meet project deadlines?

Radnia: At E Tech Group, managing timelines and resources effectively begins with detailed project planning. We break down the project into clear milestones and allocate resources based on priority and availability. Using advanced project management tools, we monitor progress in real time, allowing us to make adjustments as needed to stay on track. Regular team meetings ensure alignment, and we maintain flexibility to reassign resources if unexpected challenges arise. This proactive approach helps us consistently meet project deadlines without compromising quality.

Orth: Managing a schedule starts with writing it down and communicating it to the team. By breaking down big tasks into smaller milestones, it’s easier for the team to accomplish and easier for the PM to confirm the project is on track.

Sander: It certainly can be challenging! It’s important to keep resources (people) consistent on a project team and also make sure that each team member is hitting their billable utilization targets. The project manager needs to maintain a healthy backlog of project work while also coordinating with the client to make sure they review project deliverables in time. This way, you keep the team busy and avoid the risk of rework from getting too far ahead on deliverables before the client has had a chance to review them.

How do you handle communication and coordination between different stakeholders involved in a system integration project?

Radnia: Effective communication and coordination between stakeholders are crucial in system integration projects. At E Tech Group, we establish clear communication channels from the outset, often utilizing collaborative platforms to keep all parties informed. We hold regular meetings with stakeholders to provide updates, address concerns and gather feedback. Our project managers act as central points of contact, ensuring that everyone is aligned with the project’s goals and timelines. This structured approach minimizes misunderstandings and keeps the project moving forward smoothly.

Orth: Holding regular meetings to review status and issues keeps everyone on the same page. Following up with minutes helps drive completion of action items.

Payne: Ensuring all stakeholders are on the same page, even if that means scheudling a meeting all together, goes a long way to build a consensus for all the stakeholders, whether that is engineering, operations, maintenance, or the site leadership team. Understand that all of those groups have different needs, so the goal is to work a plan to meet all those needs.

Malyszko: Effective communication and coordination are critical to the success of any system integration project. From the outset, it’s vital to establish clear communication channels and technology preferences during the project kickoff with the customer. Each customer may have their preferred collaboration tools, and if not, we ensure a seamless collaboration experience using our own tools and methods. The rise of platforms like Microsoft Teams and Cisco WebEx, especially during COVID, has made these tools integral to day-to-day activities. It’s not just about the tools. It’s also essential to establish a regular cadence of meetings, updates and touchpoints to keep all stakeholders aligned and informed throughout the project. This approach ensures smooth collaboration, timely decision-making and the successful delivery of the project.

Sander: A solid communication plan at the start of the project goes a long way to enabling a good working relationship between all stakeholders. A key part of this plan is a detailed responsible, accountable, consulted, and informed (RACI) chart defining who is responsible for each deliverable. These should both be re-evaluated at key project milestones to keep up with any changes.

Can you share a situation where managing stakeholder expectations was particularly challenging, and how you resolved it?

Radnia: In one project, managing stakeholder expectations became challenging when a client requested last-minute changes that could have delayed the timeline and increased costs. To address this, our team at E Tech Group quickly assessed the impact of the changes and presented a clear, data-driven analysis to the client. We offered alternative solutions that aligned with their goals without compromising the project timeline. By maintaining transparency and open communication, we were able to reach a compromise that satisfied all parties and kept the project on track.

Orth: After we rolled out the first phase of a multiphase project, we learned that the operators weren’t happy with the finished product. Through a lessons-learned meeting with our customer, we determined that the operations team should have been directly represented earlier in the design process. After making this adjustment, we had much greater success with the remaining phases.

Sander: We had a project where we thought we had the right stakeholders involved early on. We completed a series of design reviews, all with minor adjustments and feedback from the client. All signs pointed to us being right on track with deliverables. Then, new stakeholders took an interest in the project and started requesting minor changes … then some larger changes to key deliverables. To resolve this, the project manager stopped work and called a meeting with all project stakeholders to discuss the risk to the budget and schedule. He eventually got a consensus on how we could implement the changes needed. Like most project issues, with a little creativity and negotiation, you can find a solution that works for everyone.

When collaborating with multiple vendors and partners during a system integration project, what are some of the common challenges you face?

Radnia: Collaborating with multiple vendors and partners during a system integration project often presents challenges such as misaligned priorities, communication gaps, and differing technical standards. At E Tech Group, we mitigate these challenges by establishing clear roles, responsibilities, and communication protocols from the start. Regular coordination meetings ensure that all parties are on the same page, and we use standardized documentation to streamline processes. Our approach fosters collaboration and ensures that all vendors and partners work toward a unified goal.

Orth: In isolation, each vendor/partner has their own way of doing things. It is essential to bring everyone together early to identify dependencies so that each party is prioritizing work on the critical path.

Payne: Communication has to be proficient among multiple vendors and partners, not just between the people working on the project. System to system communication is important within the project, so a lot of time is spent identifying the things that need to be communicated between the different systems and ensuring you get it right the first time. This will make startup more efficient. When multiple vendors and stakeholders are involved, one slip in the schedule can have a ripple effect. The better you can communicate and the better you can stay on top of any issues, the better you can maintain alignment among stakeholders.

Sander: The solution to most project challenges boils down to great communication (early and often) and risk management. This is certainly true when working with multiple parties. The project team has to have the foresight (and courage) to identify risks early in the project, find creative mitigation strategies, keep an eye out for new risks, and communicate all of this to everyone. A RACI chart in these types of projects becomes critical so everyone is on the same page on who is responsible for what.

How do you ensure smooth collaboration and integration between different third-party solutions?

Radnia: To ensure smooth collaboration and integration between different third-party solutions, E Tech Group emphasizes compatibility and interoperability from the outset. We conduct thorough compatibility assessments and engage in detailed planning with all parties involved. Our team also develops standardized integration protocols to ensure that all systems can communicate effectively. Continuous testing throughout the project lifecycle helps identify and resolve any issues early. Regular communication with third-party vendors ensures alignment and fosters a cooperative environment, leading to successful integration.

Orth: Creating an interface document that defines all the inputs and outputs ensures a smooth integration.

Malyszko: Ensuring smooth collaboration and integration between different third-party solutions often requires going beyond what’s outlined in the customer’s scope of work. It’s crucial for the system integrator to ask the right questions: Who is responsible for what? How does everyone’s system communicate? What are the bi-directional data requirements? These are details that are often assumed to be covered but typically fall on the SI to coordinate. We achieve this through collaborative working sessions with all vendors involved, where we hammer out these details in real time. Relying solely on email can lead to miscommunication. There’s nothing like face-to-face interaction, even if it’s via Microsoft Teams meetings, to ensure everyone is aligned and any potential issues are addressed upfront. This proactive approach minimizes gaps and ensures a seamless integration process.

What processes do you have in place to ensure continuous improvement in your project management practices?

Radnia: At E Tech Group, continuous improvement in project management is driven by a commitment to learning and innovation. We regularly review and analyze completed projects to identify areas for improvement. Lessons learned are documented and shared across the organization to enhance future projects. We also invest in ongoing training for our project managers, ensuring they stay updated on the latest methodologies and tools. Additionally, we solicit feedback from clients and stakeholders, which helps us refine our processes and deliver better outcomes.

Orth: Continuous improvement is an essential part of our culture. Our team members are always on the lookout for ways to do their work more efficiently. In project closing meetings, we consider what we will do differently the next time we execute a similar project, and always work to apply that going forward.

Payne: We use what we call a community of practice. It’s a small group of teammates from different disciplines of the organization that routinely review all aspects of the business to ensure our processes are still relevant and add the right value. We are always looking to improve and streamline our procedures. In addition, we have a quality process. If we find a project that didn’t go as planned, we can conduct a quality observation to investigate what happened, what in our process allowed this outcome, and what we can do process-wise and procedure-wise to prevent it from happening again.

Sander: At Vertech, we’re all about continuous improvement. But you’re not going to improve if you don’t know where you went wrong. After every project, we do lessons-learned exercises, including a full root cause analysis. Then, we take our insights and incorporate them into our ongoing continuous improvement process. The temptation is to complete a project and reassign resources right into the next project or shore up another project that’s in trouble. But, if you move too fast, you’ll miss key opportunities to improve your procedures and define best practices. Don’t let this happen!

Figure 5: WM partnered with Vertech to implement an Ignition SCADA system from Inductive Automation at the WM landfill in Alberta, Canada. Courtesy: Vertech

Can you provide an example of a significant improvement made to project management approach based on past experiences?

Radnia: Based on past experiences, we identified the need for greater flexibility in our project management approach, which led us to adopt a hybrid methodology. Previously, we struggled with rigid timelines that couldn’t easily accommodate changes. By integrating agile practices into our traditional project management framework, particularly in the commissioning and installation phases, we improved our ability to adapt to client needs. This shift resulted in smoother project execution, quicker delivery times, and higher client satisfaction.

Payne: Historically, we had approached project management with a one-size-fits-all solution. As the scale of our projects has grown, we realized one size does not fit all. We updated our procedures to allow for a small project execution philosophy and a large project execution philosophy. Our initial philosophy was geared toward large scale project execution. Typically, we complete a lot of smaller projects and those benefit from a small project methodology.

What strategies do you use to explain technical concepts to non-technical stakeholders?

Radnia: When explaining technical concepts to non-technical stakeholders, we at E Tech Group focus on clarity and relevance. We use analogies and simple language to relate complex ideas to everyday experiences. Visual aids like diagrams and flowcharts help illustrate how systems work. We also prioritize discussing the practical impact—how the technology will benefit the stakeholder’s specific needs or goals. This approach ensures that non-technical stakeholders can make informed decisions and feel confident in the project’s direction.

Orth: A picture is worth a thousand words, so simple flow diagrams and user interface mockups are key to engaging non-technical stakeholders.

Sander: Most of our clients are fairly technical, at least enough to understand what we’re doing and the effort involved. I think the key here is to use plain language. Avoid acronyms and other unnecessary technical jargon. It also helps to look at project scope and design from the perspective of the control system Users. We often encourage our engineers to “think like a plant operator” during testing and design.

System integrator questions on IT/OT integration, cybersecurity

System integrators help with integration of information technology and operational technology devices and systems with attention to cybersecurity.

What are the major risks associated with system integration projects, and how do you mitigate them?

Radnia: The major risks associated with system integration projects include:

• Data breaches and unauthorized access: Integrating systems can expose vulnerabilities. This can happen if there are no access controls or encryption. We mitigate this by using multi-layered security. It includes firewalls, intrusion detection systems, and role-based access control (RBAC).
• Incompatibility and interoperability issues: Different systems may not connect. This can create security gaps or inefficiencies. To mitigate this, we assess compatibility and use middleware to bridge gaps between systems.
• Legacy systems vulnerabilities: Legacy systems often lack modern security. They are vulnerable to attacks. We use network segmentation and virtual patching to protect these systems from threats.
• Data integrity and loss: During integration, there is a risk of data corruption or loss. To prevent this, we back up data, validate it, and use redundancy.

Orth: Safety and security: Ensuring that customers are implementing risk assessments and providing that information to the design and delivery teams.

Compressed timelines: The system integrator is often the last person on the floor for project installations. Delays in install often do not translate to extension of startup dates. Mistakes can happen when we are rushing to make the schedule. Having a customer that understands the issue and risks is crucial.

Sander: From an IT/OT perspective, device and system compatibility issues can make integration difficult. Cybersecurity vulnerabilities can also arise, especially when you’re connecting legacy systems to modern networks. To address these challenges, we perform rigorous testing in a controlled environment to find and resolve compatibility issues. We also complete comprehensive risk assessments to identify cybersecurity vulnerabilities early in the project. Hardening legacy systems, employing network segmentation, vetting third-party solutions, implementing access control, and data encryption are essential.

What security challenges do you face during system integration, and how do you mitigate them?

Radnia: The primary security challenges during system integration include:

• Data transfers: Integrated systems’ data transfers are at risk of interception and tampering. We mitigate this by using secure protocols like TLS/SSL and implementing data encryption both at rest and in transit.
• With multiple integrated systems, managing user and device authentication is complex. To fix this, we use centralized identity management and MFA to secure access.
• Real-time monitoring and incident response: It’s tough to detect and respond to security incidents in integrated systems. We mitigate this by using a unified security platform. It provides real-time visibility and automated incident response.

Orth: Many of our customers do not yet have a mature OT cybersecurity process. We often find ourself in a competitive situation where security was omitted from the scope of work, requiring us to get creative in delivering solutions which are not inherently vulnerable. Our security team is engaged and watching the threat landscape to implement hardening in design and configuration to the best of our ability within scope and budget.

When it comes to cybersecurity, what remains the biggest hurdle for system integrators?

Radnia: The biggest hurdle for system integrators in cybersecurity is balancing security and uptime. Robust security measures are crucial. But they must not disrupt operations, especially in OT environments where downtime is costly. This needs careful planning. It requires deploying effective, low-impact security solutions. Also, we must keep stakeholders updated to align security and operational goals.

Orth: The products from most control system vendors do not yet have security built-in and enabled by default. Cybersecurity can be complicated and is a rapidly changing field. Specialized resources are often required who understand both IT and OT systems and what the motivations are for both sides. Finding the people who can excel at this can be challenging.

Malyszko: The biggest hurdle for system integrators in cybersecurity is ensuring that the customer’s security requirements are understood and addressed from the outset. Often, the cybersecurity and identity groups within IT on the enterprise side are not the ones driving requirements for Industrial Control Systems (ICS); however, they find themselves responsible for hardening these systems after they’re deployed. As an SI, we can take a proactive approach by involving IT early and often, which minimizes the need for rework later. This approach counters the old mindset that IT is just going to get in the way. By advocating for our customers and integrating IT and cybersecurity representation into the process from the beginning, we help prevent costly and time-consuming issues down the road. It’s much easier to address security upfront than to clean up after a cybersecurity incident.

Sander: The biggest hurdle we face is balancing security with operational requirements. Industrial control systems environments require high availability and real-time processing, which can be at odds with cybersecurity measures that introduce latency or require downtime for updates. We take a risk-based approach that prioritizes security measures based on the criticality of the systems and the potential impact of a breach. Layered security (defense in-depth) can provide protection without compromising operational efficiency. We also collaborate with the client’s OT team to ensure that our cybersecurity measures align with their needs and don’t disrupt critical processes.

What is your process for detecting and responding to cybersecurity incidents in integrated systems?

Radnia: Our process for detecting and responding to cybersecurity incidents involves several key steps:

• Continuous monitoring: We use advanced tools to detect threats. They monitor network traffic, system logs, and user behavior in real time.
• Incident detection and analysis: After detecting an anomaly, we quickly analyze it. We use security information and event management (SIEM) tools and threat intelligence feeds to determine the incident’s nature and extent.
• Containment and mitigation: If an incident is confirmed, we will isolate affected systems to prevent further spread. We then apply mitigation techniques such as patching vulnerabilities or restoring systems from backups.
• Root cause analysis and recovery: After containment, a thorough investigation finds the root cause. Then, long-term solutions are applied to prevent recurrence.
• Post-incident review and improvement: Finally, we review our response to the incident. We will update our response plan as needed.

Orth: First you have to be looking to know if an incident has occurred or is occurring. We collect vital network and server log information and scrub that against known indicators of compromise, tactics techniques and procedures, or known exploited vulnerabilities. Leveraging machine learning for deviation from normal in OT environments is critical to providing secure solutions when working with systems that were not designed with security at the forefront.

Malyszko: Our process for detecting and responding to cybersecurity incidents in integrated systems begins with building OT security software directly into the solution. This specialized software allows us to inspect and baseline ICS traffic, helping to answer critical questions: What assets are out there, are they up to date, and what systems should be communicating with each other? By establishing this baseline, we can identify any deviations that might indicate a security incident.

These alerts can be seamlessly integrated into the customer’s security information and event management (SIEM) system, ensuring that the IT and cybersecurity teams are immediately informed of potential threats. However, detecting anomalies is just the first step. It’s equally important to provide education on what these alerts mean, helping cybersecurity professionals understand the context and intent behind the traffic. As integrators, we play a crucial role in bridging the gap between OT and IT, guiding teams on what matters most in the ICS environment to ensure that they can respond effectively to any incidents.

Sander: We use real-time monitoring tools such as SIEM systems that are designed to detect anomalies and threats in industrial environments. Likewise, intrusion detection systems detect unauthorized access or unusual activity. Behavior analytics can help detect deviations from normal operational patterns, which may indicate a cybersecurity incident. If a threat is spotted, we have a comprehensive plan that outlines the steps to be taken, including containment, eradication, recovery and communication protocols. We also conduct a forensic analysis of the event as well as a post-incident review.

How do you ensure data consistency, accuracy and integrity throughout the integration process?

Radnia: We ensure data consistency, accuracy, and integrity through:

• Data validation and verification: We use automated checks to ensure data consistency and integrity during system transfers.
• Redundant storage and backup: We use redundant storage and keep backups to recover data if it is lost or corrupted.

Orth: We assign one of our principal engineers to each project who is responsible for the technical execution against the customers’ requirements and Eosys philosophy of technical excellence. This assures the consistency of delivery across all phases of a project even when it is broken up into multiple delivery teams due to size or timeline. Additionally, our project archives and development machines are backed up on a regular basis to ensure integrity and prevent loss of progress.

How has IT/OT integration changed the way system integrators do their work in a good and bad way?

Radnia: Efficiency and cybersecurity can improve though complexity and risk may increase.

Good: Improved efficiency: IT/OT integration allows real-time data flow. It improves decision-making and boosts efficiency.

Improved security: Unified security frameworks boost monitoring and response across IT and OT.

Bad: Increased complexity: Integrating IT and OT systems makes management and security more complex. Each environment has unique requirements and challenges.

Higher risk exposure: Integration expands the attack surface. It exposes OT systems, which often lack strong cybersecurity, to cyber threats.

Orth: The Good: adoption of best practices and cybersecurity solutions from IT teams that have been doing this for years. The Bad:  IT/OT integration has exposed systems which were never designed for security to threats that did not exist when they were deployed. Additionally, it is difficult to communicate unique needs of OT networks (reliability, latency and jitter) to security teams who have no good frame of reference of what operational environments are like.

How do you incorporate emerging technologies like AI and machine learning into your cybersecurity strategy?

Radnia: We incorporate AI and machine learning into our cybersecurity strategy by:

Automated threat detection: Use AI tools to analyze network traffic and user behavior. They can find anomalies that may indicate threats.

Predictive analytics: Using machine learning to predict and prevent security incidents based on past data.

Incident response automation: Use AI to automate routine tasks in incident response. This includes isolating affected systems and blocking malicious IPs.

Continuous learning and adaptation: Machine learning models learn from new data. This improves their accuracy and reduces false positives over time.

Orth: We’ve been using an AI enabled network detection tool since long before AI was at the forefront of solutions in IT and OT Cybersecurity. AI/ML enables reduction in time to discovery of a compromise, and helps to aggregate information from multiple sources to minimize false positives allowing our very lean IT and cyber departments to keep up with the alerts which occur from our monitoring tools. We are currently investing in ways to leverage cloud-based solutions to improve our ability to detect adversarial activity and further reduce the number of false positives which our team must follow up on.

Sander: We can enhance cybersecurity by leveraging AI or machine learning for anomaly detection, predictive analytics, and automated incident responses. AI can enable automated responses to certain types of incidents, reducing the time to mitigate threats. The flip side is that AI and machine learning models require high-quality data to function effectively. Poor or biased data can lead to false positives/negatives. It can also be a challenge to integrate AI into existing cybersecurity infrastructure.

What are your thoughts on the future of cybersecurity in the context of system integration?

Radnia: Cybersecurity in system integration will become more automated. There will be a greater use of AI and machine learning for proactive threat management. A shift toward zero-trust architectures will also occur. As integration grows, especially with IoT devices, cybersecurity must evolve. It must address new vulnerabilities. This will require more advanced tools and better collaboration between IT and OT teams.

Orth: With manufacturing a prime target of ransomware groups our customers are taking OT Cybersecurity more seriously. We believe everyone should be paying more attention to a comprehensive, adaptable and prioritized framework to enhance OT cybersecurity. This approach requires a level of understanding in OT that not everyone has yet grasped, but it is foundational for effectively securing operations. If companies targeted by OT cyber-attacks had implemented such a framework, almost all known OT attacks would have been mitigated. EOSYS has always been an early adopter of building customer awareness of this type of functionality, and on the leading edge of implementing this approach.

Malyszko: The specialized nature of industrial cybersecurity (ICS), with its critical requirements for availability and determinism, means that traditional IT security approaches often fall short. Looking ahead, the future of cybersecurity in system integration will require SIs to take a more proactive approach, even when cybersecurity isn’t explicitly outlined in the scope of work. By integrating security considerations from the start, SIs can help prevent vulnerabilities that could compromise mission-critical operations. This proactive stance not only enhances the security posture of ICS environments but also elevates the level of service SIs provide to their customers, ensuring that current and future systems are robust against evolving cyber threats.

Sander: We’ll probably see more automation in cybersecurity processes, driven by AI and machine learning, to handle the growing complexity and volume of threats. We’ll also see a shift toward zero trust architecture, where every access request is verified. Finally, blockchain may play a larger role in ensuring data integrity and securing transactions in integrated systems. Of course, as cybersecurity defenses improve, attackers will also evolve, requiring constant innovation in the space. This comes with more regulations around cybersecurity and data protection, so integrators will need to stay up-to-date and compliant.

How do you address cybersecurity concerns related to the internet of Things (IoT) within integrated systems?

Radnia: To address IoT-related cybersecurity concerns, we:

• Isolate IoT devices from critical IT and OT systems. This will contain potential breaches.
• Use strong authentication: Use strong methods to protect IoT device communications.
• Regularly update all IoT devices with the latest security patches.
• Use IoT-specific security tools: Use tools made to monitor and protect IoT devices from unique threats.

Orth: We are excited about the possibility of deploying zero trust architectures where it makes sense in OT and the manufacturing space. Places like digital transformation or IoT that live at the edge are prime candidates for deploying zero trust. Applying zero trust at the gateway level could also be the game-changer that allows this technology to proliferate to the manufacturing floor.

How do you balance the need for security with maintaining system performance and usability?

Radnia: We balance security with system performance and usability by:

• Deploying lightweight security solutions: Use security tools that are fast and low-resource.
• Adaptive security controls: Use adaptive security controls. They should adjust to the current threats. Do this without affecting system operations.
• Regular performance monitoring: Continuously monitor system performance. Identify security measures that may hurt usability. Adjust configurations as needed.
• User training and awareness: Teach users security best practices. This will reduce the risk of security incidents that could affect performance.

Orth: Unfortunately, cybersecurity will never be frictionless. The key is to design security in as early in the process as possible by working with the eventual end users and system suppliers to assure that the security controls implemented will not impede the system’s ability to operate. By leveraging passive solutions when possible, and taking advantage of the physical security that is often available in the manufacturing environment, you can design solutions that are secure, but functional.

What are the key cybersecurity challenges you encounter during the system integration process?

Radnia: Key cybersecurity challenges during system integration include:

• Compatibility and interoperability: Ensure that different systems integrate without creating vulnerabilities. They have varying security postures and protocols.
• Legacy systems: Protecting older systems without modern security features. They must be integrated with newer, more secure platforms.
• Data integrity and privacy: Ensuring data is accurate and private when transferring and storing it across integrated systems.
• Managing access and authentication: Secure access to integrated systems. Keep operations efficient.

Orth: Key challenges include but are not limited to: 1) The limitations of support for critical cybersecurity controls natively within the different systems deployed in industrial control systems. 2) Desire of ICS system users to not be restricted by the implementation of security controls. 3) The lack of maturity, awareness and understanding of the risk to operations to cybersecurity impacts in operational technology.

What measures do you take to ensure that your team is trained in the latest cybersecurity practices?

Radnia: To ensure the E Tech Group team is trained in the latest cybersecurity practices we:

• Provide continuous education: Hold regular training on the latest in cybersecurity. Cover trends, tools, and threat intelligence.
• Encourage certifications and courses: Encourage team members to pursue recognized cybersecurity certifications and courses.
• Run simulated exercises: Run regular penetration tests and red team simulations. They will boost skills and find areas to improve.

Foster cross-disciplinary collaboration with teamwork between IT and OT teams. They should share knowledge and stay updated on new threats and best practices.

Orth: We are a member of the MFG-ISAC providing intelligence on the ever-changing cybersecurity landscape. We also stay engaged with CISA to understand what they are recommending both in employee development as well as currently known exploited vulnerabilities. We are also engaged with Idaho National Lab (INL) and are looking for ways to train our team on cyber informed engineering (CIE) to use good engineering design process to minimize the impact of cybersecurity failures in operating environments.

Malyszko: We ensure our team is trained to recognize cyber threats like phishing, ransomware and malware, which can cause catastrophic damage to our business and our customers. We’ve adopted a zero-trust approach for ICS, meaning no system is trusted by default, unlike the old days when controls were open for easy connectivity. This shift is crucial for securing mission-critical systems. We also rely on top ICS and Industrial IT vendor solutions such as deep packet inspection and secure remote access to enhance cybersecurity, ensuring our team and our systems are protected against evolving threats.

Sander: We have ongoing training programs that cover the latest cybersecurity threats, technologies and best practices. These include both formal training sessions and hands-on exercises. Team members are encouraged to obtain industry-recognized certifications like CISSP, CISM or GICSP. We also conduct cybersecurity simulations and drills to prepare for real-world scenarios. We try to foster a culture of learning with opportunities for webinars, conferences, and professional networks. We also provide training that covers both IT and OT cybersecurity to ensure that team members understand the unique challenges and requirements of securing an industrial control system environment.

Mark T. Hoske is editor-in-chief, Control Engineering, WTWH Media, mhoske@wtwhmedia.com.

CONSIDER THIS

How can you improve your control and automation system integration projects?

ONLINE

This system integrator roundtable article was mentioned in the Global System Integration Report, distributed in the December  2024 Control Engineering and Plant Engineering issues.

Original content can be found at Control Engineering.

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our WTWH Media editorial team and getting the recognition you and your company deserve. Click here to start this process.