Five Questions with Stephen Dubuisson

Troubleshooting issues and correcting problems that arise in operating critical infrastructure is a demanding job – everything is high priority. Every day is a new challenge.

For utility-scale electric resources, maintaining market viability requires a team that understands the assignment – like the engineers in the Trimark Operations Center (TOC) – and a proven leader like Stephen Dubuisson. As TOC’s Support Manager, Stephen works on Remote Intelligent Gateways (RIGs), revenue meters, SCADA systems (PPC, HMI, Historian), meteorologic stations, routers, firewalls, field network enclosures – basically every system power generation and battery storage resources need to maintain compliance, ensure data integrity, and deliver pinpoint control.

Stephen started at Trimark as an Operations Support Technician and quickly advanced to lead the TOC’s engineering services team. On any given day, his job may require technical leadership, analytical thinking, problem solving and collaboration – exactly the sort of range required to keep critical resources humming along.

To better understand how he operates, we asked Stephen to answer five questions about his work:

1. Is there a typical day in the TOC? What would that look like?

Stephen Dubuisson (SD): TOC personnel expect every day to be atypical. We run all sorts of issues to ground to resolve problems and prevent them from happening again. This week, we supported CAISO meters and RIG compliance; adjusted firewall rules for new devices; identified why an inverter wouldn’t respond to SCADA instructions; and determined why resources weren’t following their ADS (Automated Dispatch System) and AGC (Automatic Generation Control) instructions.

There are some constants though. We take pride in providing excellent technical support and customer service; just like we expect in our personal lives. Clients need answers now. So we answer our customer-support line before it goes to voicemail or to an automated menu and our TOC Engineers provide same-day response to email inquiries. One thing you can always count on? We’ll be there to help our customers.

2. What are the most common issues you see? When a customer says “Our RIG is down!” what’s the first thing you check?

SD: The most common service tickets involve telemetry for CAISO RIGs and revenue meters. RIGs and meters are required for market participation and provide critical information to balance the grid. So the TOC takes a proactive approach to monitoring these systems and get an early warning if something goes wrong. We typically know within five minutes when a resource loses their Energy Communications Network (ECN), ANIRA, or public internet supported telemetry. That’s a big headstart compared to CAISO’s non-compliance notification system which typically notifies site owners after the resource has been offline for 30-minutes. Since ADS is required for every resource in the CAISO market, five minutes of lost production matters. Battery Energy Storage System (BESS) resources often have an even more critical timeline and opportunities to participate in more markets. Because BESS often provides AGC response to support grid stability, every second of downtime matters!

“The best emails I receive are the ones that communicate a sigh of relief.”

We monitor every RIG and router covered by a TOC Service Agreement. Using results from monitoring tests, we resolve issues quickly, expedite remote support, escalate when necessary, and recommend corrective actions to on-site Operations and Maintenance teams.

3. What are some proactive measures you apply to expedite problem identification, replacement, software/firmware updates, and other corrections?

SD: Our real-time monitoring system lets Trimark notify clients about lost telemetry before CAISO sends their notice. That’s valuable time that owners can use to fix an issue and/or file an exemption to avoid penalties. Because data is sent over a private network, resource owners don’t have visibility on telemetry status outside their systems. Trimark fills that gap by providing real-time status. That’s an irreplaceable benefit for critical power generation and battery storage resources.

My personal favorite proactive measure is our QA process which we redesigned last year. Much of that credit goes to Ian Robinson for developing and enforcing a process that ensures our customers receive projects that operate the latest versions of Trimark’s software; reviews site documentation to confirm accurate as-builts for TOC support; and meets the high standards our customers expect. The process provides an important failsafe that supports operational integrity for critical power generation infrastructure.

4. Tell me about a time when Trimark’s Operation Center saved the day for one of our customers.

SD: Recently, the California Independent System Operator (CAISO) directed a customer to stop generating power at a cluster of nine PV and BESS resources generating nearly 550 MW. This effectively suspended them from the market.

The instruction was sent because values for compensated reactive power (UGMV) appeared to oscillate. That’s a big problem because reactive power is required to stabilize grid voltage.

While the systems were configured and deployed by others, Trimark’s TOC has a service contract to support their RIGs, meters, and SCADA systems. To identify the source of the problem, TOC engineers met with the O&M provider, asset management group, SCADA support, and a CAISO-certified Inspector. Within 24 hours, Trimark discovered that the actual power characteristics were stable. The problem was that the real-time data sent to CAISO was wrong. That said, inaccurate data can be almost as big of a problem for balancing authorities as actual oscillation.

The TOC advised the client to request a test window to validate our hypothesis. By the end of day two, TOC engineers confirmed that the “oscillation” was, in fact, limited to telemetry and metering and that the resources were not introducing oscillations to the grid. Ultimately, the TOC verified that the meter program (configured by others) was inadequate for the peer-to-peer metering and associated losses in the complex resource cluster. Our testing produced the proof required to explain the issue to CAISO and justify returning the resources to normal operation.

“Our real-time monitoring system lets Trimark notify clients about lost telemetry before CAISO.”

The real problem wasn’t just perceived oscillation. Rather, the issue was that the sites lost revenue because resources in the cluster were curtailed and couldn’t sell power. So our TOC engineers worked with the asset management team to expedite returning the resource to production. About 50 hours after the original event, the resources were approved for market participation potentially saving the site owners millions of dollars in lost revenue.

The best emails I receive are the ones that communicate a sigh of relief. After the event, Trimark was immediately contracted to reprogram, test, and certify new meter programs. Just another day in the office, or in the field, for the TOC.

5. How do you spend your time when you’re not supporting critical electrical infrastructure?

SD: In the last year I’ve picked up reading, which I had previously only ever known for academic purposes. In a world full of distractions, being able to focus on a single task is a perfect escape. My daughter Isabela was born almost three years ago. Being her dad is an adventure. There’s nothing in the world quite like hearing her laughter or seeing her smile. She creates toddler-friendly chaos and brings me joy unlike anything else.

Trimark’s Operation Center at a Glance

The TOC helps resource owners operate effectively, avoid downtime and optimize revenue for renewable resources like photovoltaic (PV), battery (BESS), wind, biomass, and geothermal as well as for legacy generation including Liquified Natural Gas (LNG), hydroelectric, and nuclear.

That might mean 24/7 site monitoring, equipment maintenance or meter recertification. Or it could require complex coordination with CAISO involving an exemption or outage request. It might a friendly voice to walk you through a simple password reset. Or it might require expediting a replacement when devices reach end of life. Regardless of what you need to keep generating, the TOC is here to help.