How one plant survived the blackout

At 3 a.m. on Friday, August 15, Joe Torr knew he was in for a tough day. Arriving long before dawn at Delphi Corp.'s massive Flint East manufacturing complex in Michigan, where he is site facility engineer and energy coordinator, Torr and his team faced the aftermath of the biggest power failure in the nation's history.

11/01/2003


At 3 a.m. on Friday, August 15, Joe Torr knew he was in for a tough day. Arriving long before dawn at Delphi Corp.'s massive Flint East manufacturing complex in Michigan, where he is site facility engineer and energy coordinator, Torr and his team faced the aftermath of the biggest power failure in the nation's history.

The cascading failure of the power grid serving the northeastern United States and southern Canada the previous afternoon had caused wild swings in voltage but little damage beyond shutting down some equipment at the automotive parts manufacturing facility. But Consumers Energy, the local electric utility, had warned that Friday might bring a different story.

When Torr made his first call to the utility at that predawn hour, his worst expectations were confirmed. With a hot summer's day expected and power plants still down across the grid, Consumers Energy predicted it would have to shut off all power to the Delphi site unless it could achieve a major reduction in energy consumption.

By the end of the day, thanks to a web-based power monitoring system, Delphi had reduced its power draw on the utility from a typical day's 25 MW to slightly more than 4 MW. More important, the organized shutdown Torr and his team achieved left equipment at Flint East unscathed. "With computers running everything today, you can do serious damage to electronic equipment if it's not turned off in an orderly fashion," said Torr.

Friday, 8 a.m.

"Once everyone got in to work, we asked them to reduce all noncritical items like air conditioning and lighting," explained Laura Bricault, utility monitoring system administrator for the Delphi East facility. "Our consumption actually peaked at 8 a.m. that day at 24 MW.

"We started noticing a significant curtailment in usage by 10 a.m. when we got down to 21 MW," she said. Bricault was able to watch consumption dropping from the computer on her desk, which is connected over Delphi's intranet to the facility's power monitoring system.

The system's monitors constantly report power consumption by equipment throughout the 4.6 million sq ft manufacturing complex. The power monitors are connected to more than 120 devices — from electrical transformers and substations to other heavy energy users, such as air compressors, water towers, chillers, and electric furnaces. Each piece of equipment is vital to Delphi's manufacturing processes.

"With more than 100 buildings at the Flint East site, it's impossible to keep track of our power system manually," said Torr. "Our web-based system lets us keep tabs on our electric consumption in real time. Being able to report actual consumption to Consumers Energy throughout the day is what made an orderly shutdown possible."

Friday, 11:30 a.m.

"Consumers Energy had a conference call at 11 a.m. with all their power plants, so when I called in at 11:30, they were pretty sure they would have to black us out by 1 p.m.," said Torr. He consulted with Flint East's site manager and plant managers, and then began an orderly shutdown.

"They looked at the manufacturing schedules for the various plants and decided that the oil filter plant could be shut down immediately, since their manufacturing schedule was not as tight as some of the other facilities," explained Torr. "Although we were already down to 16 MW, we also decided to cancel the entire second production shift for the entire complex."

Since it was just an hour before the shift was to begin at 2 p.m., many of the workers had to be met at the gate and told to go home.

Friday, 1:45 p.m.

"We were now down to 12.2 MW, about 50% of normal consumption, but the utility was asking for more" said Bricault. "I have a virtual setting for each plant on my computer, showing what equipment is located in each facility. After looking at what was still on, we called each of the plants, telling them to shut down all the process, and most facility equipment."

Less than an hour after the directive was given, Flint East was using just 7 MW of electricity.

"We saw a glitch at 2:45 p.m. when the monitoring system detected that an air compressor had accidentally been turned back on, but as soon as it was turned off we went back down to 7 MW," said Torr.

As Bricault continued to monitor the energy consumption charts on her computer screen and identify the power usage of every substation on the site, calls were placed to each plant manager to look for additional equipment that could be turned off. "At 5:15 p.m. we were down to 4.1 MW," she said.

Friday, 5:30 p.m.

"At 5:30 Consumers Energy called to say the crisis was over," said Torr. "It had started raining in the western part of the state, reducing power demand as the ambient temperature cooled. The utility had also begun to receive more power from the grid. We could begin turning equipment on again."

Now the Delphi team faced an even bigger challenge: getting everything back up and running in time for the Saturday morning shift, when they needed to make up for the lost production.

"People started coming in around 10 p.m. to restart the air compressors, which are centrally controlled and take a long time to get back up to the right pressure," said Torr. "But each individual plant had to organize its own powerup.

"A lot of behind-the-scenes equipment has to be turned on before you can begin production, like water cooling towers, air conditioning that's critical to different areas, and the electric furnaces used to make spark plugs, which have to be brought back up to the right temperature for the manufacturing process."

The Flint East complex ran at about 60% of typical consumption on Saturday, and by Monday morning everything was back to normal.

"When we bought the power monitoring system, we never had this kind of scenario in mind," said Torr. "We've had curtailments before, but they've never told us we could be completely shut down."

Like most companies that install a power monitoring system, Delphi's primary goal was to reduce energy costs. As more plant engineers at the site recognized the value of the information being generated, additional monitors were added to equipment beyond the electrical distribution system, such as the compressed air system.

"The site has five air compressors, which represent 10% of our total electrical usage," Bricault explained. "By monitoring them, we found that we needed only two compressors on the weekends. We also hooked up the electric furnaces [to the monitoring system] and found we could occasionally dial them back to a lower temperature setpoint to reduce energy costs. Next, we plan to add the water cooling towers, which are powered by big 150%%MDASSML%%250-hp motors."

The power monitoring system has also proven its usefulness beyond the electrical system. "We extended it to natural gas last year," explained Bricault. "With prices rising rapidly, we've spent more through June this year than our entire natural gas bill in 2002. Obviously, we're actively looking for new ways to reduce our gas consumption."

Although the power monitoring system is used primarily to monitor energy consumption, Torr and his team are also exploring ways to use it to bring more equipment, such as lighting, under central control. "We could make load shedding and shifting much faster that way and cut energy costs even more," he explains.

Torr says the ability to monitor and control power is more important than ever. "The world has changed a lot in the last few years for many reasons, and the power supply could be more erratic in the future. There's a great deal of uncertainty now about power — where you're buying it from, the stability of the power grid, and the reliability of local supplies when all the different power systems are tied together. There's more vulnerability than ever before."

Putting the power monitoring system on the web was a critical turning point for Delphi. Currently, key personnel — from individual maintenance people to plant managers — can view the same information simultaneously, allowing them to make critical decisions and take action quickly.

Delphi's Flint East complex has been a real utility-monitoring innovator — never more so than on August 15, 2003.

Editors note: Information for this column was provided by Square D/Schneider Electric, provider of the PowerLogic power monitoring system used by Delphi.





Top Plant
The Top Plant program honors outstanding manufacturing facilities in North America.
Product of the Year
The Product of the Year program recognizes products newly released in the manufacturing industries.
System Integrator of the Year
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
November 2018
2018 Product of the Year finalists, mild steel welding: finding the right filler, and new technique joins aluminum to steel.
October 2018
Tools vs. sensors, functional safety, compressor rental, an operational network of maintenance and safety
September 2018
2018 Engineering Leaders under 40, Women in Engineering, Six ways to reduce waste in manufacturing, and Four robot implementation challenges.
October 2018
2018 Product of the Year; Subsurface data methodologies; Digital twins; Well lifecycle data
August 2018
SCADA standardization, capital expenditures, data-driven drilling and execution
June 2018
Machine learning, produced water benefits, programming cavity pumps
Summer 2018
Microgrids and universities, Steam traps and energy efficiency, Finding help with energy projects
October 2018
Complex upgrades for system integrators; Process control safety and compliance
November 2018
Analytics quantify processes, Fieldbus networking and IIoT, Choosing the right accelerometer

Annual Salary Survey

After two years of economic concerns, manufacturing leaders once again have homed in on the single biggest issue facing their operations:

It's the workers—or more specifically, the lack of workers.

The 2017 Plant Engineering Salary Survey looks at not just what plant managers make, but what they think. As they look across their plants today, plant managers say they don’t have the operational depth to take on the new technologies and new challenges of global manufacturing.

Read more: 2017 Salary Survey

The Maintenance and Reliability Coach's blog
Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
One Voice for Manufacturing
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Maintenance and Reliability Professionals Blog
The Society for Maintenance and Reliability Professionals an organization devoted...
Machine Safety
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
Research Analyst Blog
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Marshall on Maintenance
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
Lachance on CMMS
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.
Material Handling
This digital report explains how everything from conveyors and robots to automatic picking systems and digital orders have evolved to keep pace with the speed of change in the supply chain.
Electrical Safety Update
This digital report explains how plant engineers need to take greater care when it comes to electrical safety incidents on the plant floor.
IIoT: Machines, Equipment, & Asset Management
Articles in this digital report highlight technologies that enable Industrial Internet of Things, IIoT-related products and strategies.
Randy Steele
Maintenance Manager; California Oils Corp.
Matthew J. Woo, PE, RCDD, LEED AP BD+C
Associate, Electrical Engineering; Wood Harbinger
Randy Oliver
Control Systems Engineer; Robert Bosch Corp.
Data Centers: Impacts of Climate and Cooling Technology
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
Safety First: Arc Flash 101
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
Critical Power: Hospital Electrical Systems
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
Design of Safe and Reliable Hydraulic Systems for Subsea Applications
This eGuide explains how the operation of hydraulic systems for subsea applications requires the user to consider additional aspects because of the unique conditions that apply to the setting
click me