How one plant survived the blackout

At 3 a.m. on Friday, August 15, Joe Torr knew he was in for a tough day. Arriving long before dawn at Delphi Corp.'s massive Flint East manufacturing complex in Michigan, where he is site facility engineer and energy coordinator, Torr and his team faced the aftermath of the biggest power failure in the nation's history.

11/01/2003


At 3 a.m. on Friday, August 15, Joe Torr knew he was in for a tough day. Arriving long before dawn at Delphi Corp.'s massive Flint East manufacturing complex in Michigan, where he is site facility engineer and energy coordinator, Torr and his team faced the aftermath of the biggest power failure in the nation's history.

The cascading failure of the power grid serving the northeastern United States and southern Canada the previous afternoon had caused wild swings in voltage but little damage beyond shutting down some equipment at the automotive parts manufacturing facility. But Consumers Energy, the local electric utility, had warned that Friday might bring a different story.

When Torr made his first call to the utility at that predawn hour, his worst expectations were confirmed. With a hot summer's day expected and power plants still down across the grid, Consumers Energy predicted it would have to shut off all power to the Delphi site unless it could achieve a major reduction in energy consumption.

By the end of the day, thanks to a web-based power monitoring system, Delphi had reduced its power draw on the utility from a typical day's 25 MW to slightly more than 4 MW. More important, the organized shutdown Torr and his team achieved left equipment at Flint East unscathed. "With computers running everything today, you can do serious damage to electronic equipment if it's not turned off in an orderly fashion," said Torr.

Friday, 8 a.m.

"Once everyone got in to work, we asked them to reduce all noncritical items like air conditioning and lighting," explained Laura Bricault, utility monitoring system administrator for the Delphi East facility. "Our consumption actually peaked at 8 a.m. that day at 24 MW.

"We started noticing a significant curtailment in usage by 10 a.m. when we got down to 21 MW," she said. Bricault was able to watch consumption dropping from the computer on her desk, which is connected over Delphi's intranet to the facility's power monitoring system.

The system's monitors constantly report power consumption by equipment throughout the 4.6 million sq ft manufacturing complex. The power monitors are connected to more than 120 devices — from electrical transformers and substations to other heavy energy users, such as air compressors, water towers, chillers, and electric furnaces. Each piece of equipment is vital to Delphi's manufacturing processes.

"With more than 100 buildings at the Flint East site, it's impossible to keep track of our power system manually," said Torr. "Our web-based system lets us keep tabs on our electric consumption in real time. Being able to report actual consumption to Consumers Energy throughout the day is what made an orderly shutdown possible."

Friday, 11:30 a.m.

"Consumers Energy had a conference call at 11 a.m. with all their power plants, so when I called in at 11:30, they were pretty sure they would have to black us out by 1 p.m.," said Torr. He consulted with Flint East's site manager and plant managers, and then began an orderly shutdown.

"They looked at the manufacturing schedules for the various plants and decided that the oil filter plant could be shut down immediately, since their manufacturing schedule was not as tight as some of the other facilities," explained Torr. "Although we were already down to 16 MW, we also decided to cancel the entire second production shift for the entire complex."

Since it was just an hour before the shift was to begin at 2 p.m., many of the workers had to be met at the gate and told to go home.

Friday, 1:45 p.m.

"We were now down to 12.2 MW, about 50% of normal consumption, but the utility was asking for more" said Bricault. "I have a virtual setting for each plant on my computer, showing what equipment is located in each facility. After looking at what was still on, we called each of the plants, telling them to shut down all the process, and most facility equipment."

Less than an hour after the directive was given, Flint East was using just 7 MW of electricity.

"We saw a glitch at 2:45 p.m. when the monitoring system detected that an air compressor had accidentally been turned back on, but as soon as it was turned off we went back down to 7 MW," said Torr.

As Bricault continued to monitor the energy consumption charts on her computer screen and identify the power usage of every substation on the site, calls were placed to each plant manager to look for additional equipment that could be turned off. "At 5:15 p.m. we were down to 4.1 MW," she said.

Friday, 5:30 p.m.

"At 5:30 Consumers Energy called to say the crisis was over," said Torr. "It had started raining in the western part of the state, reducing power demand as the ambient temperature cooled. The utility had also begun to receive more power from the grid. We could begin turning equipment on again."

Now the Delphi team faced an even bigger challenge: getting everything back up and running in time for the Saturday morning shift, when they needed to make up for the lost production.

"People started coming in around 10 p.m. to restart the air compressors, which are centrally controlled and take a long time to get back up to the right pressure," said Torr. "But each individual plant had to organize its own powerup.

"A lot of behind-the-scenes equipment has to be turned on before you can begin production, like water cooling towers, air conditioning that's critical to different areas, and the electric furnaces used to make spark plugs, which have to be brought back up to the right temperature for the manufacturing process."

The Flint East complex ran at about 60% of typical consumption on Saturday, and by Monday morning everything was back to normal.

"When we bought the power monitoring system, we never had this kind of scenario in mind," said Torr. "We've had curtailments before, but they've never told us we could be completely shut down."

Like most companies that install a power monitoring system, Delphi's primary goal was to reduce energy costs. As more plant engineers at the site recognized the value of the information being generated, additional monitors were added to equipment beyond the electrical distribution system, such as the compressed air system.

"The site has five air compressors, which represent 10% of our total electrical usage," Bricault explained. "By monitoring them, we found that we needed only two compressors on the weekends. We also hooked up the electric furnaces [to the monitoring system] and found we could occasionally dial them back to a lower temperature setpoint to reduce energy costs. Next, we plan to add the water cooling towers, which are powered by big 150%%MDASSML%%250-hp motors."

The power monitoring system has also proven its usefulness beyond the electrical system. "We extended it to natural gas last year," explained Bricault. "With prices rising rapidly, we've spent more through June this year than our entire natural gas bill in 2002. Obviously, we're actively looking for new ways to reduce our gas consumption."

Although the power monitoring system is used primarily to monitor energy consumption, Torr and his team are also exploring ways to use it to bring more equipment, such as lighting, under central control. "We could make load shedding and shifting much faster that way and cut energy costs even more," he explains.

Torr says the ability to monitor and control power is more important than ever. "The world has changed a lot in the last few years for many reasons, and the power supply could be more erratic in the future. There's a great deal of uncertainty now about power — where you're buying it from, the stability of the power grid, and the reliability of local supplies when all the different power systems are tied together. There's more vulnerability than ever before."

Putting the power monitoring system on the web was a critical turning point for Delphi. Currently, key personnel — from individual maintenance people to plant managers — can view the same information simultaneously, allowing them to make critical decisions and take action quickly.

Delphi's Flint East complex has been a real utility-monitoring innovator — never more so than on August 15, 2003.

Editors note: Information for this column was provided by Square D/Schneider Electric, provider of the PowerLogic power monitoring system used by Delphi.





The Top Plant program honors outstanding manufacturing facilities in North America. View the 2015 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
Doubling down on digital manufacturing; Data driving predictive maintenance; Electric motors and generators; Rewarding operational improvement
2017 Lubrication Guide; Software tools; Microgrids and energy strategies; Use robots effectively
Prescriptive maintenance; Hannover Messe 2017 recap; Reduce welding errors
The cloud, mobility, and remote operations; SCADA and contextual mobility; Custom UPS empowering a secure pipeline
Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Mobility as the means to offshore innovation; Preventing another Deepwater Horizon; ROVs as subsea robots; SCADA and the radio spectrum
Research team developing Tesla coil designs; Implementing wireless process sensing
Commissioning electrical systems; Designing emergency and standby generator systems; Paralleling switchgear generator systems
Natural gas engines; New applications for fuel cells; Large engines become more efficient; Extending boiler life

Annual Salary Survey

Before the calendar turned, 2016 already had the makings of a pivotal year for manufacturing, and for the world.

There were the big events for the year, including the United States as Partner Country at Hannover Messe in April and the 2016 International Manufacturing Technology Show in Chicago in September. There's also the matter of the U.S. presidential elections in November, which promise to shape policy in manufacturing for years to come.

But the year started with global economic turmoil, as a slowdown in Chinese manufacturing triggered a worldwide stock hiccup that sent values plummeting. The continued plunge in world oil prices has resulted in a slowdown in exploration and, by extension, the manufacture of exploration equipment.

Read more: 2015 Salary Survey

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.
The maintenance journey has been a long, slow trek for most manufacturers and has gone from preventive maintenance to predictive maintenance.
Featured articles highlight technologies that enable the Industrial Internet of Things, IIoT-related products and strategies to get data more easily to the user.
This digital report will explore several aspects of how IIoT will transform manufacturing in the coming years.
Maintenance Manager; California Oils Corp.
Associate, Electrical Engineering; Wood Harbinger
Control Systems Engineer; Robert Bosch Corp.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me