Doing a Double-Take: SCADA replication software refines Carlsberg plant operations

At the Leeds, U.K. brewing & bottling plant of Danish brewing group Carlsberg, an asynchronous fail-over and replication solution from Double-Take Software is keeping production going when server failures take critical systems off-line.

05/09/2008


At the Leeds, U.K. brewing & bottling plant of Danish brewing group Carlsberg , an asynchronous fail-over and replication solution from Double-Take Software is keeping production going when server failures take critical systems off-line.
Brewing three million hectoliters of beer each year—including the company’s popular British brand, Tetley's—and operating 24 hours a day, server failure used to hit production hard.
Over the last five years, explains Senior Systems Engineer Graeme Walker, server failure has knocked out the brewery’s SCADA system twice. On each occasion, he relates, it has taken IT engineers between two and four hours to replace the hardware and reinstate the system. And with its SCADA systems inactive, brewing operations would grind to a halt—with a four-hour interruption to brewing the equivalent of a loss in production of approximately 190,000 liters of beer.
In mid-2007, explains Walker, the decision was made to upgrade the plant’s Intellution SCADA to a more current version—now sold and supported under the GE Proficy banner. But while this contained an improved fail-over capability, the extent of the data replication wasn’t total. Recognizing that the upgrade would require new servers, Carlsberg began to explore other options.
"Given that the fail-over mechanism didn’t meet our requirements, we decided to look at an external solution,” says Walker. “We initially considered a Microsoft cluster, but it seemed very complex and the cost was prohibitive. Instead, Polestar , our IT partner, suggested we take a look at the data replication and fail-over capabilities of Double-Take Software."
A test system was set up to see if the Double-Take solution would deliver the required fail-over capability. A model of the proposed infrastructure was built by establishing an ethernet connection between a production unit on the plant floor and a "target" server located elsewhere in the facility, to which data would be replicated. Using a variety of methods to recreate realistic failure scenarios, Carlsberg engineers then caused the production server to fail—monitoring whether the fail-over process worked, and if all the required data was present.
“In every test, the Double-Take solution failed over and failed back seamlessly. After the trials were completed, we had no hesitation in giving Polestar the green light to implement the full solution,” recalls Walker.
It was an evaluation process that was fairly typical, adds Double-Take’s Worcester, U.K.-based Sales & Marketing Director, Ian Masters. Some of the company’s largest customers are in the manufacturing industry, he notes, with a common requirement being a low server load.
“With asynchronous replication, there’s no real performance impact: The CPU load is just 3 or 4 percent,” says Masters. “Asynchronous replication doesn’t guarantee total replication. Over IP networks, packet loss can happen, but the data on the target system will be usable,‘crash-consistent’ data. We’re hardware- and application-agnostic, and all the customer needs is a Windows operating system.”


Carlsberg Senior Systems Engineer Graeme Walker says asynchronous fail-over and replication software from Double-Take is keeping production going when server failures take critical systems off-line.


Carlsberg’s server upgrade took place in January, relates Walker, with the SCADA upgrade following immediately. From a previous installation that he characterizes as “15 servers and no redundancy”, the plant now has five primary SCADA servers; two secondary servers used for server databases, pictures, and recipes; and two servers dedicated to the role of replication target servers.
And in addition to their back-up role in the event of server failure, the replication servers also offer operational benefits, he adds. Server maintenance, for example, has been greatly simplified.
“If we need to investigate problems, install patches, or implement server or software upgrades, we manually fail over to the target units, leaving us free to work on the production servers without compromising performance,” says Walker. “Another benefit of having two identical datasets is that we are able to run our tape backup from the target servers. This means that the back-up process doesn't interfere with production performance, and we run the backups during standard working hours.”
Carlsberg recently had genuine need to call on its new fail-over capability—just weeks after going live. “Some plant engineers accidentally cut the power supply to a production server,” says Walker. “On the shop floor, the operators didn’t notice a thing.”





No comments
The Top Plant program honors outstanding manufacturing facilities in North America. View the 2013 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
The Leaders Under 40 program features outstanding young people who are making a difference in manufacturing. View the 2013 Leaders here.
The new control room: It's got all the bells and whistles - and alarms, too; Remote maintenance; Specifying VFDs
2014 forecast issue: To serve and to manufacture - Veterans will bring skill and discipline to the plant floor if we can find a way to get them there.
2013 Top Plant: Lincoln Electric Company, Cleveland, Ohio
Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Plant Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.

Bring focus to PLC programming: 5 things to avoid in putting your system together; Managing the DCS upgrade; PLM upgrade: a step-by-step approach
Balancing the bagging triangle; PID tuning improves process efficiency; Standardizing control room HMIs
Commissioning electrical systems in mission critical facilities; Anticipating the Smart Grid; Mitigating arc flash hazards in medium-voltage switchgear; Comparing generator sizing software

Annual Salary Survey

Participate in the 2013 Salary Survey

In a year when manufacturing continued to lead the economic rebound, it makes sense that plant manager bonuses rebounded. Plant Engineering’s annual Salary Survey shows both wages and bonuses rose in 2012 after a retreat the year before.

Average salary across all job titles for plant floor management rose 3.5% to $95,446, and bonus compensation jumped to $15,162, a 4.2% increase from the 2010 level and double the 2011 total, which showed a sharp drop in bonus.

2012 Salary Survey Analysis

2012 Salary Survey Results

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.