Is Big Data the means for survival in a down market?

In an uncertain market, more oil and gas producers are relying on advanced analytics to make intelligent cost-cutting decisions.

By Sidney Hill, Jr. May 31, 2016

The current slump in oil and gas prices is lending credence to the old business axiom that any down market presents an opportunity for someone. Information technology (IT) suppliers-especially those specializing big data and advanced analytics-are among the businesses seeing their prospects improve during this particular downturn, as oil and gas producers seek to plug leaks on the profit side of their balance sheets.

"A couple of years ago, we had to chase customers; now we see them reaching out to us," said Moray Laing, an energy industry consultant with SAS Institute Inc., an advanced analytics software provider.

When oil prices were flirting with the $100-per-barrel mark, a fair number of oil and gas companies showed interest in advanced analytics technology, but there didn’t seem to be a sense of urgency about adopting them. "In the upstream, in particular, they didn’t really have to worry about strategies to boost profitability," Laing said. "The money was just rolling in."

When prices dropped, Laing noticed that shell-shocked hydrocarbon producers initially stopped investing in IT altogether. "They were focused on cutting costs," he said, "but with the sustained drop in prices, we’re seeing a renewed interest in finding tools to help manage assets more profitability."

And that, according to Laing and others in the business, is where big data and advanced analytics technologies excel. A February 2015 survey conducted by IDC Energy Insights indicated oil and gas companies are turning to big data and advanced analytics to help soften the impact of falling prices. The survey queried CIOs from 20 non-OPEC oil and gas companies about their IT spending plans for 2015 and beyond. More than 85% of the CIOs reported that their IT budgets would either remain the same or increase slightly for the foreseeable future, and those who planned decreases said the cuts would amount to less than 5%.

"I was surprised by those numbers, said Chris Niven, oil and gas research director for IDC Energy Insights. "Given the trend of falling prices, I was expecting to hear about much bigger IT spending cuts."

In addition to surprising industry analysts with the sparseness of their expected budget cuts, the oil and gas CIOs indicated they are giving careful consideration to how they plan IT funding.

External data services and telecommunications infrastructure—both of which play integral roles in many corporate advanced analytics strategies—topped the list of technologies slated for budget increases going forward. Even before the terms "big data" and "advanced analytics" emerged, oil and gas companies relied on outside service firms to provide a wide range of data including geological information about oilfields, real-time pricing data, and other economic reports. It makes sense that 55% of the CIOs plan to increase spending on these services-even as oil prices continue to fall.

Meanwhile, 35% of the CIOs said they plan to increase spending on telecommunications infrastructure, which is necessary for sharing data across the oil and gas supply chain, particularly when information must get to and from isolated locations such as offshore drilling rigs or the more remote portions of developing shale plays.

Big data, a strategic IT investment

Niven views these numbers as part of a larger, industry-wide shift toward more strategic IT spending. Ultimately, Niven believes, this shift will result in widespread adoption of big data and analytics among oil and gas companies, but he thinks companies currently are taking some prudent first steps in that direction.

According to Niven, chief among those steps are:

  • Building solid IT frameworks, which is what the increased spending on external data services and telecommunications systems is designed to accomplish
  • Identifying areas in which IT is likely to produce the most value per dollar spent and thus minimize the potential drain on profits caused by falling oil and gas prices.

Niven said that in building their IT frameworks, companies are seeking a holistic view of their enterprises, which is necessary for conducting advanced analytics. "When oil prices were high, and the industry was growing rapidly, companies were not very strategic about their IT purchases," he said. "They would just slap in an application for whatever need arose at the time. If they needed to improve drilling performance, they would buy a drilling application. If they wanted to boost production, they would buy a production application. So, they wound up with a bunch of applications that work well on their own, but they don’t all interoperate. To perform true analytics, you need your technology to provide a more holistic view."

When seeking areas in which IT can produce the biggest bang for the buck in a down market, most oil and gas companies find themselves focusing on strategies for optimizing production of existing wells. The logic behind this is simple. Exploring for a new source of oil is an expensive undertaking, and one that is not likely to offer a great return on investment with prices at near-historic lows. But if you can find ways of lowering the cost of pumping oil from an already-producing reservoir, you have a fighting chance at boosting profit margins, even in a low-price climate.

While lowering the cost of pulling oil from existing wells makes sense in the current economic climate, it is not necessarily an easy task, which is why companies are showing greater interest in big data and advanced analytics. In a sense, oil and gas companies have been laying the foundation for advanced analytics for several years, as they’ve placed sensors and other monitoring devices on equipment in oil fields, along pipelines, and in refineries. 

Building the integrated oilfield

Colleen Kennedy, research analyst for exploration and production technology with Lux Research Inc., said wider adoption of big data and advanced analytics will move the industry beyond what has become known as the digital oilfield and into to the "integrated oilfield."

The digital oilfield is characterized by the widespread use of sensors that monitor equipment performance, with most analysis taking place after events occur. In the integrated oilfield, Kennedy said, more data will be captured for viewing and analysis in real time, allowing companies to be more proactive in addressing problems and developing strategies to improve operations. While some of the industry’s major players have started down this path to the integrated oilfield, Kennedy feels that most oil and gas producers have some obstacles to overcome to get there, particularly if they want to develop reliable strategies for optimizing production.

"Since prices have dropped, there has been a big focus on the wellhead, trying to optimize production, understand production curves, and being more efficient at that part of the process," Kennedy said. "In the past, we’ve seen operators use data to focus on reducing drilling days. They’ve done a good job at that. As they focus on production efficiency, they have to look at more variables. They need to look at geophysics and other factors. Understanding all of those pieces of information, and how to integrate them, is a little more challenging."

When taking on that challenge, companies often encounter what data scientists call "dark data," which is information collected in the normal course of doing business that never proves useful for any business purpose. For oil and gas companies seeking to optimize production, dark data typically consists of older data entered on manual logs or other outdated systems that can’t easily be loaded into an advanced analytics application. A common remedy to this problem is to develop a data rationalization strategy, which is a methodology for putting all data in the same standard format. But the nature of the oil and gas business can make that difficult as well.

"Particularly if you have an oil and gas firm that grew very fast and started operating in new venues very quickly, there’s likely to be some discontinuity in their data, and that makes rationalization much harder," Kennedy said. "As a result, we’re seeing the emergence of startups and new consulting firms that specialize in solving these problems. They’re saying, ‘We’ll take all of your data, structure it, weed out the bad data, and run analytics on the good data.’"

One of those startups is OAG Analytics, which was founded in 2013 to offer advanced analytics technology and services, primarily to companies operating in North American shale fields. 

Learning to trust the data

Luther Birdzell, founder and CEO of OAG Analytics, argues that any oil and gas firm that masters the use of advanced analytics can transform its data from an expense into a revenue-generating asset, giving that firm a distinct competitive advantage.

According to Birdzell, that transformation begins with the following three-step process:

  1. Managing or rationalizing all of the firm’s data to create a good foundation for analysis
  2. Applying analytics applications that allow for "separating the signal [useful information] from the noise [non-useful information]" 
  3. Operationalizing the data, or making it useful for people making business decisions.

Birdzell believes that advanced analytics is an essential tool for oil and gas producers. However, he also emphasized that this tool will be of little value unless a company’s domain experts-its geoscientists, petroleum engineers, and corporate executives-trust the data that the tool generates enough to make it part of their regular decision-making process.

Birdzell offered the following examples of OAG Analytics customers who have reached that comfort level:

  • One customer developed a data-driven approach to asset valuation that has increased the accuracy of estimating the production potential of large acreage positions [100,000 acres and above] by more than 20%.
  • Several customers devised strategies for developing acreage more efficiently through optimum well spacing.
  • Other customers are performing statistics-based completion optimization, determining what type of frack to perform, and which fluid to use to yield certain profit levels. Some of these firms have increased barrels produced per dollar deployed by more than 10%.

"Strategically, we are allowing customers to be more efficient at getting the domain experts the information they need to make more informed decisions," Birdzell said.

Though it’s a much larger company with a longer history in the space, SAS takes a similar approach to working with oil and gas companies. Keith Holdaway, an SAS energy industry consultant, spoke of working with customers in three key areas:

  1. Cleaning up data
  2. Making data easy for users to access and manage
  3. Enriching data with meta information that allows users to understand exactly what the data means.

While working on these data management tasks, SAS also tries to guide customers to identify specific business problems that can be solved through the use of their newly cleansed data. "The idea is to break down individual solutions that provide a rapid return on investment by solving a specific business problem," said Holdaway. "At the same time, we always want our customers to be guided by an enterprise-wide vision."

Logical problem solving

Developing an enterprise-wide vision allows a company to develop a roadmap for applying advanced analytics to multiple problems in a coordinated, logical-and more cost-effective-fashion. "With this approach, you can solve multiple business problems without ending up with a thousand disparate approaches," Holdaway said. "Everything works together, and at the same time, you don’t have to wait 5 years to see any real business benefit."

Petrobras, a Brazilian company with the stated goal of being among the world’s five largest integrated oil and gas companies, deploys SAS technology in this fashion. Among other things, Petrobras uses SAS advanced analytic models to identify rock breaks that will produce oil and gas. Shortly after activating those models, Petrobras set a new company record for the most oil produced in a single day-more than 2 million barrels.

Petrobras’ average production levels have grown as well, and it has added SAS technology to other areas of the business, including its executive suite. "Our people work with huge quantities of data, and the vagueness related to that data," said Olinto Gomes de Souza, a Petrobras senior geologist. "Our challenge is finding new technologies that are going to solve our problems."

That’s a challenge facing all oil and gas companies, regardless of size, particularly in the face of falling prices. Given that dynamic, oil and gas producers should at least consider this thought from Birdzell, the OAG Analytics CEO. "If you look at any two oil and gas firms operating with equivalent resources, the one that uses its data more efficiently and effectively will always have a material competitive advantage, especially in a down market," he said.

Sidney Hill is a graduate from the Medill School of Journalism at Northwestern University. He has been writing about the convergence of business and technology for more than 20 years.

Quick story synopsis


Low oil prices have prompted producers to make processes as efficient as possible, but to do this requires the proper analysis of data, which not all companies have properly ordered and stored.


Use specailized software to keep track of data and organize it accodingly.

Actions to take

Learn to trust data by following these three steps:

  1. Manage or rationalize all of the firm’s data to create a good foundation for analysis. 
  2. Apply analytics applications that allow for separating the useful information from the non-useful information.
  3. Operationalize the data, or make it useful for people making business decisions.

ONLINE extra

See additional stories from Hill linked below.

Original content can be found at Oil and Gas Engineering.