How one plant survived the blackout

At 3 a.m. on Friday, August 15, Joe Torr knew he was in for a tough day. Arriving long before dawn at Delphi Corp.'s massive Flint East manufacturing complex in Michigan, where he is site facility engineer and energy coordinator, Torr and his team faced the aftermath of the biggest power failure in the nation's history.


At 3 a.m. on Friday, August 15, Joe Torr knew he was in for a tough day. Arriving long before dawn at Delphi Corp.'s massive Flint East manufacturing complex in Michigan, where he is site facility engineer and energy coordinator, Torr and his team faced the aftermath of the biggest power failure in the nation's history.

The cascading failure of the power grid serving the northeastern United States and southern Canada the previous afternoon had caused wild swings in voltage but little damage beyond shutting down some equipment at the automotive parts manufacturing facility. But Consumers Energy, the local electric utility, had warned that Friday might bring a different story.

When Torr made his first call to the utility at that predawn hour, his worst expectations were confirmed. With a hot summer's day expected and power plants still down across the grid, Consumers Energy predicted it would have to shut off all power to the Delphi site unless it could achieve a major reduction in energy consumption.

By the end of the day, thanks to a web-based power monitoring system, Delphi had reduced its power draw on the utility from a typical day's 25 MW to slightly more than 4 MW. More important, the organized shutdown Torr and his team achieved left equipment at Flint East unscathed. "With computers running everything today, you can do serious damage to electronic equipment if it's not turned off in an orderly fashion," said Torr.

Friday, 8 a.m.

"Once everyone got in to work, we asked them to reduce all noncritical items like air conditioning and lighting," explained Laura Bricault, utility monitoring system administrator for the Delphi East facility. "Our consumption actually peaked at 8 a.m. that day at 24 MW.

"We started noticing a significant curtailment in usage by 10 a.m. when we got down to 21 MW," she said. Bricault was able to watch consumption dropping from the computer on her desk, which is connected over Delphi's intranet to the facility's power monitoring system.

The system's monitors constantly report power consumption by equipment throughout the 4.6 million sq ft manufacturing complex. The power monitors are connected to more than 120 devices — from electrical transformers and substations to other heavy energy users, such as air compressors, water towers, chillers, and electric furnaces. Each piece of equipment is vital to Delphi's manufacturing processes.

"With more than 100 buildings at the Flint East site, it's impossible to keep track of our power system manually," said Torr. "Our web-based system lets us keep tabs on our electric consumption in real time. Being able to report actual consumption to Consumers Energy throughout the day is what made an orderly shutdown possible."

Friday, 11:30 a.m.

"Consumers Energy had a conference call at 11 a.m. with all their power plants, so when I called in at 11:30, they were pretty sure they would have to black us out by 1 p.m.," said Torr. He consulted with Flint East's site manager and plant managers, and then began an orderly shutdown.

"They looked at the manufacturing schedules for the various plants and decided that the oil filter plant could be shut down immediately, since their manufacturing schedule was not as tight as some of the other facilities," explained Torr. "Although we were already down to 16 MW, we also decided to cancel the entire second production shift for the entire complex."

Since it was just an hour before the shift was to begin at 2 p.m., many of the workers had to be met at the gate and told to go home.

Friday, 1:45 p.m.

"We were now down to 12.2 MW, about 50% of normal consumption, but the utility was asking for more" said Bricault. "I have a virtual setting for each plant on my computer, showing what equipment is located in each facility. After looking at what was still on, we called each of the plants, telling them to shut down all the process, and most facility equipment."

Less than an hour after the directive was given, Flint East was using just 7 MW of electricity.

"We saw a glitch at 2:45 p.m. when the monitoring system detected that an air compressor had accidentally been turned back on, but as soon as it was turned off we went back down to 7 MW," said Torr.

As Bricault continued to monitor the energy consumption charts on her computer screen and identify the power usage of every substation on the site, calls were placed to each plant manager to look for additional equipment that could be turned off. "At 5:15 p.m. we were down to 4.1 MW," she said.

Friday, 5:30 p.m.

"At 5:30 Consumers Energy called to say the crisis was over," said Torr. "It had started raining in the western part of the state, reducing power demand as the ambient temperature cooled. The utility had also begun to receive more power from the grid. We could begin turning equipment on again."

Now the Delphi team faced an even bigger challenge: getting everything back up and running in time for the Saturday morning shift, when they needed to make up for the lost production.

"People started coming in around 10 p.m. to restart the air compressors, which are centrally controlled and take a long time to get back up to the right pressure," said Torr. "But each individual plant had to organize its own powerup.

"A lot of behind-the-scenes equipment has to be turned on before you can begin production, like water cooling towers, air conditioning that's critical to different areas, and the electric furnaces used to make spark plugs, which have to be brought back up to the right temperature for the manufacturing process."

The Flint East complex ran at about 60% of typical consumption on Saturday, and by Monday morning everything was back to normal.

"When we bought the power monitoring system, we never had this kind of scenario in mind," said Torr. "We've had curtailments before, but they've never told us we could be completely shut down."

Like most companies that install a power monitoring system, Delphi's primary goal was to reduce energy costs. As more plant engineers at the site recognized the value of the information being generated, additional monitors were added to equipment beyond the electrical distribution system, such as the compressed air system.

"The site has five air compressors, which represent 10% of our total electrical usage," Bricault explained. "By monitoring them, we found that we needed only two compressors on the weekends. We also hooked up the electric furnaces [to the monitoring system] and found we could occasionally dial them back to a lower temperature setpoint to reduce energy costs. Next, we plan to add the water cooling towers, which are powered by big 150%%MDASSML%%250-hp motors."

The power monitoring system has also proven its usefulness beyond the electrical system. "We extended it to natural gas last year," explained Bricault. "With prices rising rapidly, we've spent more through June this year than our entire natural gas bill in 2002. Obviously, we're actively looking for new ways to reduce our gas consumption."

Although the power monitoring system is used primarily to monitor energy consumption, Torr and his team are also exploring ways to use it to bring more equipment, such as lighting, under central control. "We could make load shedding and shifting much faster that way and cut energy costs even more," he explains.

Torr says the ability to monitor and control power is more important than ever. "The world has changed a lot in the last few years for many reasons, and the power supply could be more erratic in the future. There's a great deal of uncertainty now about power — where you're buying it from, the stability of the power grid, and the reliability of local supplies when all the different power systems are tied together. There's more vulnerability than ever before."

Putting the power monitoring system on the web was a critical turning point for Delphi. Currently, key personnel — from individual maintenance people to plant managers — can view the same information simultaneously, allowing them to make critical decisions and take action quickly.

Delphi's Flint East complex has been a real utility-monitoring innovator — never more so than on August 15, 2003.

Editors note: Information for this column was provided by Square D/Schneider Electric, provider of the PowerLogic power monitoring system used by Delphi.

No comments
The Top Plant program honors outstanding manufacturing facilities in North America. View the 2015 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
Safety for 18 years, warehouse maintenance tips, Ethernet and the IIoT, GAMS 2016 recap
2016 Engineering Leaders Under 40; Future vision: Where is manufacturing headed?; Electrical distribution, redefined
Strategic outsourcing delivers efficiency; Sleeve bearing clearance; Causes of water hammer; Improve air quality; Maintenance safety; GAMS preview
SCADA at the junction, Managing risk through maintenance, Moving at the speed of data
Safety at every angle, Big Data's impact on operations, bridging the skills gap
The digital oilfield: Utilizing Big Data can yield big savings; Virtualization a real solution; Tracking SIS performance
Applying network redundancy; Overcoming loop tuning challenges; PID control and networks
Driving motor efficiency; Preventing arc flash in mission critical facilities; Integrating alternative power and existing electrical systems
Package boilers; Natural gas infrared heating; Thermal treasure; Standby generation; Natural gas supports green efforts

Annual Salary Survey

Before the calendar turned, 2016 already had the makings of a pivotal year for manufacturing, and for the world.

There were the big events for the year, including the United States as Partner Country at Hannover Messe in April and the 2016 International Manufacturing Technology Show in Chicago in September. There's also the matter of the U.S. presidential elections in November, which promise to shape policy in manufacturing for years to come.

But the year started with global economic turmoil, as a slowdown in Chinese manufacturing triggered a worldwide stock hiccup that sent values plummeting. The continued plunge in world oil prices has resulted in a slowdown in exploration and, by extension, the manufacture of exploration equipment.

Read more: 2015 Salary Survey

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.
This article collection contains several articles on the vital role of plant safety and offers advice on best practices.
This article collection contains several articles on the Industrial Internet of Things (IIoT) and how it is transforming manufacturing.
This article collection contains several articles on strategic maintenance and understanding all the parts of your plant.
click me