It’s been more than a month since vandals severed AT&T’s fiber optic line in a remote area south of Ukiah, sending untold tens of thousands of customers into an information blackout. (“Untold” because AT&T isn’t telling. A spokesman said hard numbers might encourage more vandalism.)
The outage affected wireless and wireline service in Mendocino, Humboldt and Del Norte counties, knocking out at least seven 911 call centers, disabling credit card machines and ATMs and bringing many government offices and local businesses to a standstill.
Many doctors’ offices, pharmacies and medical clinics were unable to access computerized patient records, make phone calls or contact specialists. For example, a man suffering a medical crisis couldn’t fax test results to his specialist in Portland, nor could he reach the doctor by phone.
In the days after the outage, locals recounted the impacts on Facebook. A local tech company employee was talking to a potential client when the phone cut out mid-call. “It’s hard to be competitive and close deals when phone lines are down,” he wrote.
A woman said her friends had trouble getting food since EBT and credit card systems weren’t functioning.
Librarian Ruth Moon noted, “Students taking online courses were unable to access course materials, delaying their completion of homework and assigned readings.”
“Imagine the millions of dollars of productivity we lost yesterday on the North Coast due to a lack of redundancy,” wrote local attorney Bill Barnum.
An emergency medical dispatcher for Southern Trinity Area Rescue was unreachable at 5 p.m., when she was scheduled to go on-call. Unbeknownst to her, there was a gap in on-call services while the clinic tracked down another volunteer dispatcher, who was on Verizon. “We have no way of knowing if any parties were unable to contact the clinic or STAR [Southern Trinity Area Rescue] number with an emergency,” the woman wrote.
The blackout was expensive, dangerous and more than a little frustrating. Most of us here in Humboldt County thought the days of such debilitating internet and phone outages were behind us. We suffered a series of them last decade, including severed fiber-optic lines in 2006 and 2007 that, like this recent outage, crippled not only the internet but also phone service, credit card machines and ATMs.
Back then, the vast majority of broadband internet in the county got routed through a single fiber optic “trunk line” running north-south, which meant that every time the line got severed — be it by an errant backhoe bucket or a massive mudslide — darkness descended.
But in late 2011, the community’s cries for broadband redundancy were answered when a company called IP Networks finished construction of a second “trunk” line running east-west, a project that cost $14.4 million, including almost $5.8 million from the taxpayer-funded California Advance Services Fund (CASF).
Or at least we thought our cries for redundancy had been answered. Turns out it’s more complicated than that.
Having two separate fiber-optic arteries does not automatically mean redundancy, which is why that word irks some of the area’s more techno-savvy residents. The second line merely created another pathway, increasing our region’s route diversity and allowing for more redundancy potential. But if your personal internet service provider isn’t leasing space on at least two routes, or if you’re not buying service from two different providers using two different routes, or if your service doesn’t have the infrastructure for automatic failover, then you don’t have redundancy. (Even then, if there’s an outage closer to your location — in the so-called “middle mile” or “last mile” of your connection — your service will go down.)
Suddenlink customers weren’t affected in last month’s outage, and a spokesman said the company leases space on both the north-south line and the east-west line. “We have taken steps that would allow us, in many instances, to avoid customer service interruptions,” he said.
Local company 101Netlink offers redundancy because it leases space on the east-west line and uses microwave infrastructure to route traffic south to another east-west line that connects in Ukiah.
But here’s where things get murky. AT&T supposedly has built-in redundancy, too. The company told this reporter more than three years ago that it was leasing space on the east-west route, presumably as backup for connections through the north-south route that it owns. AT&T boasts of its “redundant fiber network” in advertising and informational materials, and indeed, as the Outpost reported during this latest outage, some AT&T customers weren’t affected at all.
Why? That’s a tough question to answer since all telecom companies treat the details of their infrastructure as proprietary information. Despite repeated attempts to get details from AT&T, a spokesman remained vague about many aspects of the situation.
“Our network is designed to provide back-up service wherever possible for all customers, taking into account factors such as geography, the route data must travel, and age of existing facilities,” the company’s official statement read. “In rare instances, customers with large or sensitive facilities may opt to purchase back-up fiber service where available for added redundancy.”
Is that the key? Were the only AT&T customers left standing the ones who purchased backup fiber service? Some businesses and agencies, including Redwood Capital Bank and Humboldt State University, have taken measures to ensure coverage during such outages, purchasing space on dedicated backup routes. Homes and businesses in the vicinity of such big-time customers often benefit because they’re served by the same circuits.
Follow-up emails to AT&T were answered by Steven Ramirez, a spokesman for a PR agency called Beyond the Arc, which handles AT&T’s public relations in northern California and Nevada. “For this specific cable that was cut,” Ramirez wrote, “we know some circuits served by the damaged fiber had a backup route where others did not.”
Customers wanting added protection, Ramirez said, can buy local access “or even dedicated end-to-end routing engineered to specific needs.” How much does that cost? “These are typically custom options and priced according to the design and the service provided.”
How can individual customers and businesses know whether or not they’ll be protected during the next outage? Ramirez declined to get specific. “[W]e do not have a customer-by-customer breakdown and do not share specific sensitive network infrastructure details publicly.”
Earlier this week, AT&T submitted an official report to Humboldt County supervisors, recounting certain aspects of the outage and restoration, which, judging by the report, was a harrowing experience indeed.
“AT&T work crews overcame significant obstacles posed by the remote location of the damage, including cold weather and the presence of rattlesnakes,” the statement recounts. Despite these dangers, the workers bravely forged ahead.
“The nearest road to the area is a twenty to thirty minute walk away. Technicians teamed up with a local fire department’s railroad trolley operator, who used the adjacent railway to haul in power generators, fiber fusing machines, building cases for splicing, a backhoe, and all the new fiber that would be needed for repairs.
“With the necessary equipment in place the crew began splicing the 96 fibers of the severed cable and worked through the night using light and power from generators while fending off against rattlesnakes and without any space heaters. By sunrise the following morning, the crew had restored most services and a relief crew arrived to complete the remaining work.”
(Read the entire nail-biting account here.)
It took a little more than 24 hours to restore all wireline services. Wireless services weren’t fully restored until 2:30 Friday afternoon. Remote fiber line switches in Hopland and Eureka were isolated, leaving customers in those areas with dial tones but unable to make or receive phone calls outside their local areas, including calls to 911.
“The Hopland switch was restored at on 9:55 p.m. Thursday and the Eureka switch was restored at 8:45 a.m. on Friday,” the report states.
AT&T defends its record of reliability in the report, noting that during 2014 the company received an average of just two trouble reports per 100 lines in Humboldt and Mendocino counties, “well within the CPUC’s guidelines.”
CPUC is the California Public Utilities Commission, which has also been rather short on information about the outage. Spokesman Christopher Chow told the Outpost in an email, “We are not an active participant in the criminal investigation that caused the specific outage in question.”
The agency monitored AT&T’s efforts to restore service, and it has received more detailed information from the company in the month since. But it’s not releasing that information publicly. “Details about the incident and its specific impacts were provided by AT&T to the CPUC as confidential information,” Chow said.
Why would that information be confidential? After all, it was given to a regulatory government agency charged with serving the public interest. “The information is confidential because the investigation is still going on,” Chow said. “You should ask AT&T.”
Where does that leave customers? It depends on their size and finances. Last year the county commissioned a report that spelled out the local broadband situation and, in retrospect, could have served as a warning for last month’s outage. Prepared by Tina Nerat of NeraTech, the report, which can be read online here, outlined a range of concerns raised by local tech support companies. Among them: “Businesses need to do more contingency planning for emergencies. Few companies have business continuity plans, hot sites, or alternative bandwidth.”
The report also noted, “Complacency has increased about the need to install backup bandwidth now that there are multiple middle mile routes out of the region.” In other words, many people just assumed they had backup bandwidth since the technology now exists here.
Larger businesses and government agencies like Redwood Capital Bank and HSU may have set up dedicated symmetrical services that offer an extra degree of protection, but, the report notes, “Many local businesses cannot afford to purchase backup bandwidth in case of disasters.” Plus, some of the county’s more remote areas simply don’t have the option of multiple providers.
Seth Johannessen, owner of 101Netlink, said that, to a large extent, local businesses get what they pay for. “If Internet connectivity is mission critical or really valuable, you need two connections, and you need hardware installed by a network management company,” he said.
That way, if one line goes down, you have the option of a backup line and the failover system to connect you to that second line automatically. Johannessen’s company offers that, and he suggested customers call their own provider to ask specific questions about their systems.
It’s also important to consider the reliability of each internet service provider and the response time during outages, Johannessen said.
The Mendocino County Sheriff’s Department continues to investigate the vandalism that led to last month’s outage. AT&T officials will soon deliver a report to Humboldt County supervisors, according to 1st District Supervisor Rex Bohn. State Senator Mike McGuire called for “a long-term fix to ensure that there is redundant 911 emergency phone service throughout California.” And Connie Stewart, coordinator for the Redwood Coast Connect Consortium, is working with regional partners to advocate for greater broadband deployment, adoption and policy in the region.
For now, it’s clear that our redundancy celebrations were premature. Humboldt County still has a ways to go before achieving reliable and truly redundant broadband access for the majority of residents.
Be First to Comment