Companies
Fire in the Santa Cruz Mountains! Cruzio and the Loma Fire
By Peggy Dolgenos (Cruzio Internet, President) and Alana Matthews (Cruzio Marketing Associate)
October 27, 2016 — Santa Cruz, CA
(Photo above: Eagle watches Loma Fire. All photo credits: Chris Frost, Alison Lowenthal, Colin Miller, and Cruzio security camera.)
Cruzio and the Loma Fire: Day One
September 26, 2016, was a hot day in Santa Cruz. Summertime hot. Fire weather.
At about 2:45 pm, one of our ham radio enthusiasts emailed us: “Fire in the Santa Cruz Mountains. Just a quarter acre, someone’s trailer caught fire,” he told us.
In an hour the fire had spread to 200 acres. Local TV news streamed a live feed, which we tweeted: a helicopter circling Loma Prieta mountain, watching the fire race uphill.
Small planes from Cal Fire dumped loads of flame retardant on a mountain covered with dry, brittle trees and brush. At first, Cruzio staff watched like anyone else would watch, worrying about the people, pets and animals who might be affected. But it got more personal as the hours passed.
A telecommunications facility, used by many companies including Cruzio, sits on top of Loma Prieta. We serve some fiber-to-wireless customers from that site, and it’s an important backup to our fiber connections, protecting our network from outages. We’ve spent years acquiring and deploying expensive equipment there.
Starting at about 2:45 pm, we watched live TV footage of the fire approaching the bunker-like buildings and steel tower housing our equipment.
Cruzio has security cameras up there — we mostly use them to monitor weather, so we can make sure snow and ice don’t interfere with equipment in winter. The cameras can be swiveled, zoomed and aimed remotely, so we can see all around. Often, one of our technicians will spot mountain lions or eagles close to the buildings and they’ll email out screen shots. We love the cameras.
By 3:30 pm, our security cameras were picking up nearby flames.
At Cruzio headquarters, we anxiously watched the live video. We hoped the fire would miss our facility. Winds were blowing away from the buildings — we could see embers blowing away from the camera. But the situation changed by the minute.
As we swiveled the camera, we saw fire in dozens of spots burning just down the hill.
If fire had engulfed the mountaintop facility, and burned the bunkers down, our equipment would have been destroyed. But the site is prepared for fire, with a wide dirt clearance, paved perimeter road and concrete buildings. And once we saw the fire moving south, we were relieved that the fire wouldn’t destroy our equipment. Now we worried about electricity.
Cruzio and the Loma Fire: Night One
Cruzio staff were glued to our video, watching the Loma Fire burn close to our hilltop equipment for hours. By early evening, the tall flames were very near our buildings, but luckily never crossed the paved perimeter.
They came awfully close, though. As we swiveled and zoomed our camera remotely, this is what we saw:
Loma fire from Cruzio security camera, 8:24 pm, September 26, 2016.
Loma Fire from Cruzio security camera, 8:11 pm, September 26, 2016.
At about 6 pm, PG&E had cut power for safety reasons. You don’t want live electrical wires on burning poles.
How We Prepare for Emergencies Like the Loma Fire
Now back up a step, to Cruzio’s long years of planning. In 2009, when we started using mountaintop facilities, we had limited power backup at those sites. When there was a problem with power — which happens more often than you’d like, — we’d have to rush up to fix the situation.
One of Cruzio’s principles of operation is that we must do absolutely all we can to provide 24/7 service without fail. Plus, we hate going up to mountains in bad weather, in the middle of a freezing night.
So Cruzio has invested heavily in long-lasting power backups for mountaintops.
Now, we have backup batteries rated to last 24 hours. We refresh our batteries on a regular schedule every year so they are always in good shape. We regularly test our generators and monitor our power equipment. Our teams drill for emergencies and every member of our staff is ready to help. It all comes in handy at a time like this.
Those powerful batteries meant we maintained service all night while the fire blazed around the hilltop. The batteries also kept the cameras going so we continued to watch, transfixed, throughout the evening.
But after a day, we knew our batteries would run down. We needed to get up the mountain with a generator to recharge them — PG&E was estimating a whole week before power was restored.
We kept watching until late at night. Even though the fire had passed us, Cal Fire said there was no way we could get to our site until late morning. So our Field Operations staff prepared to go up first thing the next day.
Cruzio and the Loma Fire: Day Two
In the morning on September 27, 2016, Field Operations Manager Alison Lowenthal, engineer Colin Miller, and our new apprentices, Cameron Langston and Rafferty Lincoln, headed up in the truck to see if we could get to the mountaintop where fire had threatened. It was Cameron’s first day at work — what a first day! Ralph Silveira, one of our top field staff, drove out, too, but had to head home with a flat tire. Over 2,000 acres were now burning, but the fire had moved south of our site, so we were hoping to reach our equipment.
The mountain facility doesn’t have a permanent generator, so we have to bring our own when power is down. Usually we have enough battery power to last out any outage, but in this case, the power could be down for many days, PG&E said. Over a dozen nearby utility poles had burnt or fallen or both.
Cruzio needed to get the generator running to recharge the batteries before they ran out of juice.
Emergency personnel stopped our team partway up the winding mountain road. “Not safe yet.” We know there are two entrances to the site — and our customers are depending on us — so the crew drove down the mountain, onto Highway 17, and tried going in the north entrance, opposite where the fire was burning. Chris Frost, our Director of Technology and Infrastructure, coordinated with Cal Fire to determine the safest way up the mountain.
They stopped at Tomita Hill, where they took this photo of Loma:
Loma Fire September 27, 2016
The unpaved roads on that side are even more winding. Once again Field Ops were stopped by Cal Fire staff, who told them “But if you go back around the other entrance, maybe you can get in that way.” So they drove all the way back down to 17, and back up again to where they’d started. Then they waited at the road block for a few hours.
Delay was pushing the limits of our equipment. Back at Cruzio, we were watching (on internet monitors) as the power supplies drained away.
Mark Hanford, our Chief Systems Engineer, was re-routing all the customers he could away from Loma Prieta. Networks have some amount of flexibility — that’s why every site is important, and redundancy is crucial — so Mark was reconfiguring the network to route around the emergency. In a short time, he had moved almost all our customers away from the problem.
Then we waited to see if the power would go down before our crew could get in.
At 5:30 pm, half an hour was left on the battery. Time ticked away. 15 minutes. 10. At 6 pm, Mark said “Okay. Power can go off any second now.” Our team was still waiting at the Cal Fire stop, waiting for entry. Capacity on the batteries went to zero.
But miraculously our power didn’t go out. The batteries kept running. They kept going past their expected charge. Mark stayed at the office, waiting for the power to go off so he could check on his emergency network routing. 7 pm. 8 pm. The batteries didn’t die. Our guys were waiting on the mountain. 8:30 pm.
Finally, the batteries were out at 8:50. They had run on “empty” for two and half hours! [Editor’s note: Sounds like a modern-day version of the Channuka Miracle.]
Then, just after battery power went out, we got word: Cal Fire was escorting Cruzio’s team up the hill in a convoy of fire and water trucks, and onto our site. It was stop-and-go all the way up the mountain as the fire was hard to work around. Although diminished at that time, some flames were only feet away from the vehicles.
“We could hear the fire like a jet engine,” recounted Colin Miller, “You could see the embers flying into the trees [near] where the vehicles were.” It was clear, though, that the fire was moving away and our team made it up the mountain safely.
It only takes 5 minutes to power up the generator. Our crew had it going in no time. Then they took turns watching all night, and sleeping in shifts, to make sure everything ran smoothly.
Up there on the tower, above the smoke the air was clear. After a long day of braving the smoky air and fire, Field Ops took the opportunity to end the night with some stargazing. Alison Lowenthal summarized the day: “It’s intense, but it’s actually a lot of fun. I love being up on Loma.”
Cruzio and the Loma Fire: Aftermath
Cruzio went through a tense Monday, when the fire started, with flames roaring past our hilltop facility. Then all day Tuesday Cruzio staff was waiting, watching and then racing to re-charge our backup batteries before power ran out. We didn’t completely avoid an outage, but the number of people affected was small and the effect brief.
We felt good about our performance and lucky we’d escaped more damage.
By Wednesday, conditions on Loma’s hilltop had stabilized. The fire still raged south of us — 4,000 acres burning that day — but our access to the site had become pretty routine. The road block was gone. We worked with other folks on the mountaintop, sharing resources and helping each other out.
PG&E told us that power would be off for a week or more, so we settled in for the long haul. Chris Frost, our Director of Technology and Infrastructure, went on TV to explain the situation.
For several days our staff took turns going up the mountain and maintaining the generators. It was a grind, though what we went through didn’t compare to people who’d lost homes or were displaced.
A lot of fast food was consumed and a lot of time was spent looking out over the charred landscape and drifting smoke. Our team saw a dozen or more utility poles burned black and leaning, clearly a big job for PG&E.
Over the next few days, we watched fascinated as PG&E lowered pole after pole from a helicopter into place on the steep mountainside. The utility’s ground crew guided the poles into place and quickly had the new infrastructure up.
Cal Fire bravely walking toward the burn. PG&E is fixing the pole behind them.
Then, well ahead of schedule, just after noon on Monday, October 3, 2016, PG&E restored power to the mountaintop. Cruzio’s staff breathed a sigh of relief. Our saga was over for now.
A week later (10/3/16), the fire was 95% contained and people returned to their homes. In all, the fire destroyed 12 houses along with many outbuildings. Luckily, no one was injured.
Things to know about fire:
- A fire grows much faster than you’d ever imagine, especially in such dry conditions. Fire coming? Get out!
- Preparation is essential. A generous perimeter saved our gear. Our batteries and generator prevented a serious outage.
- Working together in these situations is key. All companies in the facility helped each other, making the situation much more bearable.
- PG&E really has their process down, they were extremely efficient.
- Cal Fire is a heroic organization. We owe a lot to them and we’re very grateful for all their help. Over 2,100 fire fighters came from around the state to work in challenging, steep terrain. As our staff member Colin said, “The firefighters were the most amazing thing, I just had this feeling of WOW! Being up there with them is so different from just knowing they’re up there.”
###