Troubleshooting Helps Restore Telemetry and Avoid Penalties

Whether a resource is large or small, or issues are intermittent or catastrophic, Trimark’s Operations Center helps find and fix the root cause.

When a power generation resource loses its secure data communications link with California’s balancing authority, fines can mount quickly – up to $500 for each day the failure is unreported or unresolved. So, it’s essential to know immediately when a problem occurs, request an exemption, then quickly fix the issue.

Maintaining secure data exchange is critical for power generation resources participating in the open energy market. That’s why Trimark provides service agreements designed specifically to support the telemetry requirements of the California ISO (CAISO) Business Practices Manual (BPM).

When the Trimark Operations Center noticed that our customer, an oil recycling firm in Southern California, was experiencing intermittent communication failures, our team got straight to work.

A Bit of Background

The CAISO requires generation resources to install a Remote Intelligent Gateway (RIG) and join the Energy Communications Network (ECN). Together, the RIG and ECN allow CAISO to see plant status and send secure commands – critical information the ISO needs to manage grid stability.

Because Trimark supports over 1,300 generation resources, our Operations Center can monitor the communication status for each resource in near real time. If a disruption occurs, we alert the operator immediately, often before CAISO sends a notice of non-compliance. That buys valuable time required to find and correct the issue as well as request the temporary exemption required to avoid penalties.

The Challenge

While monitoring our customer’s equipment status, we noticed when their site communications started dropping its connection five times or more every day. As a result, they were not in compliance with the ISO’s telemetry requirements. First, we alerted the owner to the situation, then helped them avoid penalties by preparing an exemption request for time to find and fix the problem.

The next step in our troubleshooting process was to determine if the root cause was caused by equipment at the site or related to the network. AT&T’s assessment showed that their circuit was stable and had not experienced an interruption for several months. AT&T suggested that we confirm that the router had a reliable power supply. With assistance from an on-site resource, we checked that theory by bypassing the uninterruptable power supply (UPS) and providing power directly to AT&T’s router. However, the problem persisted. While we had eliminated two major variables – the circuit and power – we had yet to identify the real culprit. So, we dug deeper.

All’s Well that Ends Well

The next step was to analyze event data captured by the RIG (provided and configured by others). After reviewing the RIG logs (which capture data about events like this), we eventually found that the malfunction was caused by an aging AT&T router. Like most technology, old equipment may show early warning signs of pending failure, such as frequent unprompted restarts.

With solid analytical information in hand, Trimark Operations Center team members coordinated with AT&T to replace the ECN router at the site and successfully closed the non-compliance ticket with the CAISO – thereby avoiding fines and providing the data CAISO requires to manage the grid.

About Trimark

Trimark Associates, Inc. (Trimark), the industry leader in intelligent energy control, complex metering and ISO compliance delivers industry-leading solutions to enable real-time operational control, informed management, and revenue recognition in the open energy market. Trimark’s turnkey products, engineering, and customer support services help control, measure, and manage all aspects of power production that utility-scale power producers require to maintain peak business performance.

Learn more about Trimark Operations Center and services.