Doing a Double-Take: SCADA replication software refines Carlsberg plant operations

At the Leeds, U.K. brewing & bottling plant of Danish brewing group Carlsberg, an asynchronous fail-over and replication solution from Double-Take Software is keeping production going when server failures take critical systems off-line.


At the Leeds, U.K. brewing & bottling plant of Danish brewing group Carlsberg , an asynchronous fail-over and replication solution from Double-Take Software is keeping production going when server failures take critical systems off-line.
Brewing three million hectoliters of beer each year—including the company’s popular British brand, Tetley's—and operating 24 hours a day, server failure used to hit production hard.
Over the last five years, explains Senior Systems Engineer Graeme Walker, server failure has knocked out the brewery’s SCADA system twice. On each occasion, he relates, it has taken IT engineers between two and four hours to replace the hardware and reinstate the system. And with its SCADA systems inactive, brewing operations would grind to a halt—with a four-hour interruption to brewing the equivalent of a loss in production of approximately 190,000 liters of beer.
In mid-2007, explains Walker, the decision was made to upgrade the plant’s Intellution SCADA to a more current version—now sold and supported under the GE Proficy banner. But while this contained an improved fail-over capability, the extent of the data replication wasn’t total. Recognizing that the upgrade would require new servers, Carlsberg began to explore other options.
"Given that the fail-over mechanism didn’t meet our requirements, we decided to look at an external solution,” says Walker. “We initially considered a Microsoft cluster, but it seemed very complex and the cost was prohibitive. Instead, Polestar , our IT partner, suggested we take a look at the data replication and fail-over capabilities of Double-Take Software."
A test system was set up to see if the Double-Take solution would deliver the required fail-over capability. A model of the proposed infrastructure was built by establishing an ethernet connection between a production unit on the plant floor and a "target" server located elsewhere in the facility, to which data would be replicated. Using a variety of methods to recreate realistic failure scenarios, Carlsberg engineers then caused the production server to fail—monitoring whether the fail-over process worked, and if all the required data was present.
“In every test, the Double-Take solution failed over and failed back seamlessly. After the trials were completed, we had no hesitation in giving Polestar the green light to implement the full solution,” recalls Walker.
It was an evaluation process that was fairly typical, adds Double-Take’s Worcester, U.K.-based Sales & Marketing Director, Ian Masters. Some of the company’s largest customers are in the manufacturing industry, he notes, with a common requirement being a low server load.
“With asynchronous replication, there’s no real performance impact: The CPU load is just 3 or 4 percent,” says Masters. “Asynchronous replication doesn’t guarantee total replication. Over IP networks, packet loss can happen, but the data on the target system will be usable,‘crash-consistent’ data. We’re hardware- and application-agnostic, and all the customer needs is a Windows operating system.”

Carlsberg Senior Systems Engineer Graeme Walker says asynchronous fail-over and replication software from Double-Take is keeping production going when server failures take critical systems off-line.

Carlsberg’s server upgrade took place in January, relates Walker, with the SCADA upgrade following immediately. From a previous installation that he characterizes as “15 servers and no redundancy”, the plant now has five primary SCADA servers; two secondary servers used for server databases, pictures, and recipes; and two servers dedicated to the role of replication target servers.
And in addition to their back-up role in the event of server failure, the replication servers also offer operational benefits, he adds. Server maintenance, for example, has been greatly simplified.
“If we need to investigate problems, install patches, or implement server or software upgrades, we manually fail over to the target units, leaving us free to work on the production servers without compromising performance,” says Walker. “Another benefit of having two identical datasets is that we are able to run our tape backup from the target servers. This means that the back-up process doesn't interfere with production performance, and we run the backups during standard working hours.”
Carlsberg recently had genuine need to call on its new fail-over capability—just weeks after going live. “Some plant engineers accidentally cut the power supply to a production server,” says Walker. “On the shop floor, the operators didn’t notice a thing.”

No comments
The Top Plant program honors outstanding manufacturing facilities in North America. View the 2013 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
A cool solution: Collaboration, chemistry leads to foundry coat product development; See the 2015 Product of the Year Finalists
Raising the standard: What's new with NFPA 70E; A global view of manufacturing; Maintenance data; Fit bearings properly
Sister act: Building on their father's legacy, a new generation moves Bales Metal Surface Solutions forward; Meet the 2015 Engineering Leaders Under 40
Cyber security cost-efficient for industrial control systems; Extracting full value from operational data; Managing cyber security risks
Drilling for Big Data: Managing the flow of information; Big data drilldown series: Challenge and opportunity; OT to IT: Creating a circle of improvement; Industry loses best workers, again
Pipeline vulnerabilities? Securing hydrocarbon transit; Predictive analytics hit the mainstream; Dirty pipelines decrease flow, production—pig your line; Ensuring pipeline physical and cyber security
Upgrading secondary control systems; Keeping enclosures conditioned; Diagnostics increase equipment uptime; Mechatronics simplifies machine design
Designing positive-energy buildings; Ensuring power quality; Complying with NFPA 110; Minimizing arc flash hazards
Building high availability into industrial computers; Of key metrics and myth busting; The truth about five common VFD myths

Annual Salary Survey

After almost a decade of uncertainty, the confidence of plant floor managers is soaring. Even with a number of challenges and while implementing new technologies, there is a renewed sense of optimism among plant managers about their business and their future.

The respondents to the 2014 Plant Engineering Salary Survey come from throughout the U.S. and serve a variety of industries, but they are uniform in their optimism about manufacturing. This year’s survey found 79% consider manufacturing a secure career. That’s up from 75% in 2013 and significantly higher than the 63% figure when Plant Engineering first started asking that question a decade ago.

Read more: 2014 Salary Survey: Confidence rises amid the challenges

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.