Case Study: T-Mobile Scales Digital Metering Across Data Center Portfolio

Welcome to our Case Study series, where we dive into case studies of real-life, large-scale deployments of smart building technologies, supported by the Nexus Marketplace.

I emphasize “real life” because this isn’t a marketing fluff story. We're here to share real lessons from leaders who have done the work to integrate smart building technology into their operations. I also emphasize “large scale” because we're not here to talk about pilot projects. We're here to talk about deeper commitments to changing how buildings are operated.

---

Case Study Data

• Technology Categories Mentioned: 
  
  • Device Layer: Metering - Electrical 
  • Network Layer: Network Management 
  • Application Layer: Meter Analytics, Sustainability Reporting, Capital Planning,
• Key Stakeholders: Energy Managers, Sustainability Officers, Electrical Contractors, Network Managers
• Vendors: Verdigris for electrical metering and meter analytics
• Number of Buildings: 40,000+ sensors across 100 data centers
• Project Dates: 2018 - present

---

Introduction

According to John Coster of T-Mobile, expanding a data center can cost anywhere from $10 to $20 million per megawatt. Faced with what seemed like an inevitable $11 million expansion at one facility, T-Mobile relied on its innovative use of smart building technology to uncover a surprising alternative. By deploying a cloud-connected architecture of electrical meters and analytics, an algorithm identified a misreporting meter hiding existing available capacity within the data center. This discovery saved the company from a $11 million expansion and avoided the carbon emissions that would have come with it.

This story underscores a universal truth for energy managers and capital project teams across industries: the value of trustworthy data. Energy data may be their ally, but trustworthiness is their best friend. To delve deeper into how data-driven decisions transform T-Mobile’s energy management and capital investment strategy, we spoke with John Coster of T-Mobile and Mark Chung of Verdigris, T-Mobile’s electrical metering partner. Together, they revealed how they’re building trust in their data to save millions and reduce carbon across T-Mobile’s extensive data center portfolio.

Background

John Coster is the Senior Manager of Innovation, Planning, and Strategy at T-Mobile. He’s been with the company for seven years. Coster explained, “My job is to figure out where we put, how much we put, and when we put our data centers to handle all the traffic that supports our network.”

Coster has been doing this type of work since 1988, when he helped design Microsoft’s first data center (referred to as “computer rooms” back then). Coster has seen the industry grow up through the .com boom and bust and beyond. 

T-Mobile’s business is all about monetizing spectrum—in other words, turning radio waves into revenue by enabling data exchange between devices worldwide. Cellular towers are used to do this, and T-Mobile's 100+ data centers interconnect all those cellular towers.

Coster explained how HVAC cooling and electrical consumption are directly connected to the bottom line of T-Mobile’s business: “I don't know if people know this, but for every watt that goes into a data center or goes into a computer, it is 100 percent inefficient. It gets rejected exactly the same amount of wattage and heat. So, how do we reject all that heat? And how do we do an energy mass balance calculation that says, here's how much energy is being used for our workloads, and here's how much energy is being used for the corresponding mechanical equipment?”

This is where Verdigris comes into play. Mark Chung is the CEO and co-founder of Verdigris, a California-based electrical metering device and application vendor founded in 2011. Chung started his career as an electrical engineer focused on microprocessor design. He was deeply moved by Al Gore’s film, An Inconvenient Truth, and decided to make a career pivot into using tech to improve electricity usage. 

T-Mobile has been working with Verdigris for the past seven years to meter all of its alternating-current loads within data centers, primarily consisting of computer room air conditioning (CRAC) units. Over 40,000 sensors were installed across the 100+ data centers to measure those AC loads. The primary goal of this implementation was eliminating electrical waste.

“You can’t make changes unless you have good data, and we didn’t have any good baseline data, so we started with benchmarking. Then we were able to look at where we are wasting money, energy, and carbon emissions, and how can we pull that in.”
—John Coster, T-Mobile

In addition to simply saving money, T-Mobile is a consumer-based company with more than 100 million customers, many of whom deeply care about their carbon footprint. Mitigating data center expansion means reducing multiple layers of carbon production. T-Mobile is striving to become a steward of the environment through its decisions.

Through this deep partnership, Verdigris has helped T-Mobile unlock learnings and savings opportunities they weren’t expecting. “It's always these building blocks of getting new insights because you've mined data that you couldn't have easily justified. Sometimes, [the value] is three levels in, you didn't realize that's actually what you needed. So it's a little bit of a throw it out and see what happens and go see where the value is.”

The Nexus Labs team put together the following visual of the steps that T-Mobile took to implement a trustworthy and data-driven metering program:

Technical Overview

To unlock deeper insights into their data center operations, T-Mobile needed to move beyond basic utility bills and building-level metering and gain granularity down to the CPU level. The project team began installing Verdigris power metering equipment on every circuit and every piece of air conditioning and mechanical equipment. 

Trustworthiness and Data Accuracy

Coster highlighted that the market is saturated with different telemetry data companies that can provide information, but the biggest challenge is data trustworthiness—do you really trust that the data is accurate?

“If you don’t have good analytics that allow you to see where the gaps are, you will just make bad decisions because you have high fidelity data that isn’t accurate.”
—John Coster, T-Mobile

When the project began, Coster and his team quantified their data accuracy as about 60% and set a goal to get to 90%. Verdigris supported them in achieving this goal in a handful of ways.

First, Verdigris’s cloud-connected architecture. Every device reports to the cloud, and the Verdigris platform uses a persistent hypervisor that continuously oversees all of the sensors. There is no reliance on standalone systems that must work perfectly in isolation. When all data is available in real-time and simultaneously, the platform can utilize machine learning techniques to point out anomalies, which can be used for further research and to find errors.

The second key to data accuracy was mitigating installation errors in as many ways as possible. Chung emphasized the number of field errors that occur during a large-scale metering implementation: CTs can be installed backward, wires can be crossed, phases can be mismatched, etc. Verdigris CTs are small, digitally bussed, and cannot be inserted incorrectly, which helps installers avoid these errors.

Throughout the T-Mobile project, Verdigris also improved the mobile application to support installers. The app runs through a series of checks to ensure that after installation is complete, any issues can be fixed only through remote configuration adjustments. This avoids the expense of redeploying electrical contractors to make a physical adjustment to the installation.

Beyond avoiding installation errors, Verdigris has also designed its hardware to help detect installation errors. Instead of a CT (current transformer), Verdigris utilizes hall effect sensors with an 8kHz sampling frequency, a higher sampling rate than the norm. Chung explained, “There are many ways in which the CT, the current, and the voltage are not matching correctly to give you the right power factor variance, which Verdigiris can detect with the high sampling frequency.” 

Real-World Use Cases

As Coster mentioned, data-driven insights from electrical metering often start as a game of not knowing what you don’t know. At the beginning of the project, it can be difficult to quantify the new doors of opportunity that the data will unlock for you. Coster and Chung provided us with some tangible use cases enabled by the metering implementation.

Insights Via Data Layer-Enabled Correlations

Internally, T-Mobile uses a platform called NEON (Network Engineering Ontology Tool), which serves as a holistic data layer by integrating various data sources, including CFD (computational fluid dynamics) models, building management system (BMS) data, and Verdigris metering data.

Verdigris plays a critical role by providing analytics that correlate different systems, a capability that hinges on its high sampling rate. As Chung explained, Verdigris samples every power fluctuation at 8kHz, resulting in extremely detailed historical data. When this high-resolution data is applied to aggregated loads, such as those from a DC plant, machine learning algorithms can identify patterns and relationships. These algorithms detect high-probability correlations between electrical consumption changes and specific assets or systems upstream from the electrical meter—a process known as data disaggregation.

PUE Accuracy & Saving Energy

Coster discussed the relationship between PUE (power usage effectiveness) and T-Mobile’s capital expenses. Every watt of electricity that goes into a data center gets rejected as heat, and the amount of energy it takes to offset that heat gives you the PUE. Data center managers often look at PUE to inform their decisions on future data center expansion. Typically, PUE will increase as the data center's computing capacity increases. As PUE begins to level off at a higher value, it may indicate to building owners that they are reaching the threshold of their cooling equipment’s capacity, and expansion or equipment upgrades may become necessary.

Before trustworthy granular metering was available, T-Mobile was basing its PUE on assumptions; therefore, decisions on when data center expansion was needed was reliant on these assumptions. 

Accurate PUE measures also help T-Mobile target areas of high PUE and see how it can be mitigated. “If I’m spending hundreds of millions of dollars on energy, and I can shave off 5% or 10% on my PUE, that’s huge,” explained Coster.

Outage Diagnostics

Chung and Coster recalled a major outage at one of T-Mobile’s data centers. No customers were lost, but services were impacted. The aftermath became a round of finger-pointing, and the utility provider and generator manufacturer blamed harmonic distortion issues from T-Mobile’s equipment as the cause.

Verdigris metering equipment captured the last minute of operation prior to the shutdown local to the devices. The historical data showed a high-quality sine wave, acquitting T-Mobile of the harmonic distortion accusations. 

Challenges and Lessons Learned

This case study aims to share success stories that can benefit building energy managers and capital project teams facing the challenges of understanding and lowering energy consumption throughout their portfolios. Coster and Chung shared some of the most significant challenges and lessons learned when applying Verdigris metering solutions to the T-Mobile data center portfolio.

Lesson #1: Navigating Internal Approvals and Vendor Selection

Larger corporations continually sway between seasons of capital abundance and capital scarcity. When getting a capital expense project approved, timing the seasons can be critical.

Coster credited a “perfect storm” for the right timing to get the electrical metering project off the ground at T-Mobile. First, T-Mobile had committed to major growth in the 5G network across the country, so capital was actively being allocated. Second, he noted that telemetry data at their data centers was nearly nonexistent, so the need for improvement was undebatable. 

“I had an opportunity to scrape some of the billions they were putting into 5G, and I had to do it quick,” stated Coster. 

Next came finding the right vendor. Navigating the Marketplace (aka the vendor swamp) with thousands of companies and solutions is a mind-boggling exercise we often discuss at Nexus Labs, and Coster mentioned he felt similarly, citing that choosing the right partner sometimes feels like “a leap off a cliff.” 

Given his small window, Coster wasn’t able to complete a full-blown technology readiness assessment (TRL) like many large corporations do, but he knew he had some key considerations:

• Flexibility: Given the “we don’t know what we don’t know” nature of operational data improvements like this one, Coster knew he wanted a partner who could adapt and adjust its offering as new insights were uncovered. Given that Verdigris is a relatively smaller vendor in the space, Coster had confidence that Verdigris would be able to adjust its offering on the fly and adapt to new information the project brought in.

• Scalability: While flexibility was critical, it was equally important for Coster to find a partner that could keep up with the size and speed of implementation T-Mobile needed. He needed a partner with a proven track record and implementation footprint. He needed the Goldilocks zone of flexibility and scalability.

• Internal Support Team: Coster was keen to find which groups within T-Mobile could also benefit from more granular electrical consumption data and bring them in as additional allies to support the project. For example, while Coster does capital rationalization, his operational counterparts need FDD and more accurate as-builts and could benefit as much from the metering project as he could. More internal support can exponentially increase the probability and speed of a new project.

Lesson #2: Simplify Network Management

Coster and his team at T-Mobile determined to use hardwired ethernet connections to all of the electrical meters for communications. Verdigris offers communication that is hardwired, over Wifi, or cellular. 

Coster mentioned that if he had to do it over again, he would try to make a wifi/5G solution work, even if it meant an increase in repeaters and antennas necessary. Coster encourages other energy managers to use wireless as much as possible. New security protocols made it challenging for the project team to get port assignments on a timely basis. Fewer IP addresses and fewer port assignments would save time and will be considered by Coster on future projects.

Lesson #3: Installation Inconsistency 

There are dozens of ways to wire and install an electrical meter incorrectly. Chung explained, “When we started rollout across a larger number of T-Mobile buildings, from Puerto Rico to Hawaii, that meant lots of different people would be installing the systems.” 

The project team noticed meter data anomalies that were regionally based, which indicated that installation inconsistency by certain installer groups was the root cause. 

The project team pivoted to a single professional installer across all regions and worked with the installer to create a certified professional installer process. A deep focus on the process with a single installer led to a much more rigorous post-installation checklist that allowed Verdigris to be confident that any issues would be software adjustable without a return visit. 

Lesson #4: Setting Expectations and SLAs on APIs

As the project kicked off, Chung noted that Verdigris wasn’t prepared for the amount of traffic from T-Mobile’s NEON platform requesting data, and initially, some data drops occurred. Verdigris reworked its APIs to handle the data requests. Chung stated the importance of truly understanding the customer’s use case of the API so that you can set realistic expectations within an SLA (service level agreement). 

On the flip side, Coster emphasized the importance of understanding the materiality of data sampling, which building owners need to recognize comes at a cost. For example, what is the incremental price and value of sampling once a second versus once a minute versus once every 15 minutes? T-Mobile has started to work with the term “minimum viable sampling rate” to help their team quantify this. They’ll frequently set a minimum sampling rate and test out what they can gain from higher sampling rates in select areas before implementing the higher sampling rate across their portfolio.

Conclusion

T-Mobile’s success with Verdigris highlights a critical takeaway for energy managers and capital project teams across industries: granular, trustworthy data is the foundation of informed decision-making. Whether you’re managing data centers, office buildings, or higher education campuses, the principles remain the same—accurate data enables you to uncover inefficiencies, avoid unnecessary expenses, and reduce your environmental footprint. For T-Mobile, it wasn’t just about solving a problem; it was about unlocking new opportunities by addressing what they didn’t yet know. Energy managers everywhere can see this example as a roadmap for creating smarter, more sustainable operations.