Despite IPv6 Uncertainties MSOs Are Forging Ahead

John Curran, president & CEO, American Registry for Internet Numbers

John Curran, president & CEO, American Registry for Internet Numbers

July 14, 2011 – Broadband service providers, especially on the cable side, have come a long way in preparations for the transition to a new Internet addressing system, but the scope of potential disruptions they face once the transition begins in earnest remains a disconcerting mystery.
“We know it has to be done,” says John Rubinger, director of engineering systems at Canadian MSO Eastlink. “But we won’t know what we have to deal with until we start provisioning IPv6 addresses.”

IPv6 is the name given to the next version of the IP numbering system administered globally by the Internet Assigned Numbers Authority (IANA) and regionally in North America by the American Registry for Internet Numbers (ARIN). In February IANA allocated the last of the regional blocks of 16 million IPv4 addresses available for public use, which means that, one by one, regional authorities will be running out in the months ahead.

In fact, the Asian authority, Asia-Pacific Network Information Centre (APNIC), has already depleted its store, leaving no more addresses to be allocated to service providers and other entities in that region. ARIN will run out within the next nine to 12 months, says ARIN president and CEO John Curran, at which time service providers will only be able to provision new v4 addresses out of the store they have in hand.

No one knows how many new addresses will be needed to accommodate new subscribers and independently addressed devices in an ever more connected world, but the assumption is the volume of addresses is going to explode. While the Internet Engineering Task Force (IETF) sees a need for “only” about 50 billion new addresses over the next 30 to 50 years, “long term we wanted to make sure we never have to go through another transition,” Curran says. That’s why IETF came up with a 128-bit numbering system which can support about 340 trillion, trillion, trillion addresses but is completely incompatible with the v4-based Internet.

The onus is on everyone – Website owners, ISPs, device manufacturers and enterprises and institutions as well as broadband access providers – to face up to the change at hand. Everyone, that is, except consumers. When it comes to consumers, the burden will fall on broadband providers, Curran says.

But service providers also have to worry about enterprise customers, who, in general, have been indifferent to the looming challenges posed by the v6 transition. “Very few of our enterprise customers are interested in adapting to v6,” says Frank Bulk, technology and product development manager, for Premier Communications, an Iowa-based provider of broadband over DOCSIS, DSL and fiber networks.

Curran says he’s seeing signs of proactive efforts on the part of some big corporations, but that enterprises have a long way to go. “Broadband access providers have the hardest job,” he notes. “Their customers want to access any site. If they can’t get to a site because they’re on the wrong protocol, they will think it’s the service provider’s problem. SPs have to make sure those customers can get to v6 sites. Now is the time to deploy those addresses.”

Internal Preparations

Just getting the work done from an internal perspective is hard enough. Operators must ensure every network device – routers, servers, customer premises equipment – is in sync with whatever transition modes they employ as well as with the fundamentals of working in the all-new IPv6 domain. Moreover, anything in the operations support system software stack that depends on Internet connectivity must accommodate the new address field. While v6 addressing employs the base-16 hexadecimal notation system rather than the traditional 10-base system, which cuts the character count from 128 to anywhere from 40 to 57, depending on the number of buffer characters used, this is still more than three times the space v4 address fields were designed for.

This is a problem affecting all players on the Internet. “We have a lot of code changes to make in places where v6 doesn’t fit,” notes Will Lawton, network engineer at Facebook. “We have a lot of interns working on this.”

And, for operators, the tools used to monitor performance must be adjusted. “Your software tools don’t know how to check v6,” Curran says. “You need to fix that, and you need to train your IT and operations staff on how to use these tools.”

But, as noted by Vikas Sarawat, senior architect for IP technology at CableLabs, daunting as all these things are, the cable industry has made considerable progress sorting through the challenges. “All the basic technical requirements have been taken care of,” Sarawat says.

These include support for v6 in DOCSIS 3.0 and PacketCable 2.0 specifications as well as spec enhancements for DOCSIS 2.0. CableLabs has also defined “eRouter” specification for a lightweight dual-stack cable modem router that allows a stateful DHCPv6 (IPv6 Dynamic Host Control Protocol) -provisioned device to pass both stateful and Stateless Auto Address Configuration (SLAAC) provisioning commands through to devices in accord with whichever provisioning system is required by a given device. (SLAAC, used in telco provisioning of IP addresses, depends on the host assigning itself an address that is automatically configured by router discovery, in contrast to the cable address-assigning process used with DHCP in v4 and v6).

Transition Options and Challenges

Now CableLabs is focused on helping the industry deal with the v6 migration process. Taking feedback in weekly calls with operators the organization has been documenting issues and solutions as they come up, thereby expediting the entire member community’s ability to learn from individual experiences. And CableLabs has been performing interoperability testing with vendors to ensure the network components designed to support Dual Stack operations of v4 and v6 in parallel in accord with DOCSIS 3.0 and v6-optimized DOCSIS 2.0 are working as intended.

With Comcast leading the way in a series of tech initiatives and trials, most MSOs have implemented Dual Stack capabilities in their backbones. This means v6 runs natively across those portions of the network and is translated to v4 at edge points of entry into the access networks.

Given that Dual Stack has not been implemented on the access side, with the exception of the first Dual Stack access deployments by Comcast (see February, p. 18), any operators who want to support v6-addresses at the end points must rely on a tunneling technique known as 6RD (IPv6 Rapid Deployment), which uses address translation technology at the CMTSs (cable modem termination systems) and customer premises to encapsulate v6 packets into v4 and to extract the v6 packets for transmission to v6 end points.

Once Dual Stack is operational end to end, user devices will be able to communicate in whichever protocol they’re equipped for. If a Dual-Stacked network device such as the CMTS queries the name of a destination and the DNS (Domain Name System) gives it a v4 address (a DNS A Record), the CMTS sends v4 packets. If the DNS responds with a v6 address (a DNS AAAA Record), it sends v6 packets.

Much remains to be done with respect to helping operators sort through the internal implementation issues, including refinements in procedures and identification of new tools and testing methods that can be consistently applied industry wide, Sarawat notes. “But cable operators are definitely engaged with moving ahead with v6 implementations,” he says.

The picture is different on the telco side, notes Brian Markwalter, senior vice president, research and standards at the Consumer Electronics Association. “There’s a big difference in perceptions [about v6] between cable and telecom service providers,” Markwalter says. “We need to fix that.”

“Cable does seem to be ahead of the RBOCs,” says Premier’s Frank Bulk. However, he adds, his company is encountering problems with vendor preparedness for supporting v6 on both fronts.

“Our BRAS (broadband remote access server) vendor has to do more work,” Bulk says. “Our CMTS vendor is not v6 ready either.”

Unknowns of CGN

While such problems are certain to be worked out in the near term, the larger issues service providers face have to do with the unknowns of how various factors in the transition phase to all-v6 operations will impact consumer experience. “We can’t expect subscribers to take the initiative on making v6 work,” says Eastlink’s Rubinger. “It’s our responsibility to make sure their experience is as seamless as possible.”

There’s much cause for concern as to just how heavy the weight of that responsibility will be, given all the possible glitches that could affect consumer experience once people are provisioned on v6, Rubinger says. For example, he and others note, the v4 address preservation strategy known as Carrier Grade Network Address Translation (CGN) is a solution Eastlink and virtually everyone else will have to implement, even though they have no clear idea of how negative the impact will be on consumer experience.

“I don’t see where we’ll have a choice,” Rubinger says.

That’s because v6 is not designed to work with Network Address Translation or NAT, a universally used means of providing private addresses to connected devices that run behind a publicly addressed v4 modem, which minimizes the number of public v4 address allocations. Consequently, every time a new subscriber is given a v6 address, any v4-only devices the subscriber may want to connect to the service will have to be assigned a public v4 address. Thus, the provisioning of new v6 addresses will likely do nothing to mitigate the pace of v4 exhaust and, indeed, could accelerate it, depending on the proportion of v4 devices that turn out to be in use as new subscribers come on the network.

CGN slows the pace of v4 exhaust by setting up a double level of NAT usage, where a publicly assigned v4 address on a network CGN termination point can be used to support multiple private NAT addresses to individual subscribers, and each of those subscribers can be assigned additional private NATs behind their modems to accommodate multiple additional devices. Traditional NAT operates as usual for subscribers with public v4 addresses.

CableLabs has identified the version of CGN known as NAT 444, which is in the draft standardization phase at the IETF, as one of the cable industry’s modes of dealing with the migration to v6. But, as previously reported (September 2010, p. 29), operators have long been inclined to avoid using CGN as much as possible. Now, with recognition that v4 addresses could run out long before the network makes a complete cutover to v6, the focus is on how to minimize negative impacts of CGN as well as to better understand what those impacts will be.

A series of tests CableLabs ran on CGN last year identified several points of potential disruption to network operations and consumer experience, starting with all the applications such as e-mail systems, monitoring platforms, some provisioning systems, law enforcement tracking and much else that depend on the fact that a public v4 address has traditionally been used to uniquely identify a single subscriber. While steps can be taken in the network to make sure private addresses are linked to the public addresses in these applications, the mitigation can introduce delays and even breakdowns in certain applications. For example, adaptive streaming and fast-action gaming are especially subject to delays in the CGN environment, CableLabs found.

The good news, says CableLabs’ Sarawat, is that new tests are showing that suppliers of CGN technology are finding ways to improve performance. “It’s a moving target which gives us hope things will continue to improve over time,” Sarawat says.

But there’s no sense assuming the problems will disappear, notes Chris Donley, project director for network protocols at CableLabs, which ran a demo of CGN in action at the recent Cable Show in Chicago. “We still see that v4 service running behind CGN will be a big challenge for many users,” Donley says.

“The CGN architecture limits what customers can see,” he says. “And there’s still a high probability that with NAT 444 some apps will break down. It’s a hard message for the industry. But if you want to ensure customers have a great experience on the Internet, the solution is to move to v6 as fast as possible.”

Part of the solution lies in gaining a better understanding of the routing issues associated with CGN, says CEA’s Brian Markwalter. “Some companies doing video streaming haven’t bought into CGN as a viable solution,” he says. “But we don’t think it’s fundamentally broken. We just need to get the facts and to know what’s out there that CE companies can participate in.”

Web Site Preparations

On the flip side of the dual stack there’s also a big concern over whether Web sites will be equipped to support access to content by users with v6 addresses. ARIN’s Curran says that while this was his most worrisome concern in the past, he’s less worried today owing to the progress he’s seeing with the movement of sites to supporting v6 along with v4. “I think we’ll see 90 percent of content over the next three to four years reachable by v6,” Curran says.

But, he adds, that still leaves 10 percent of Web sites in the long tail of legacy operations which will not be accessible to v6-addressed devices. “So we will have to gateway it,” he says, in reference to the translation techniques that can be used in networks to connect v6 devices to v4 content. “We’ll have five years of Dual Stack, and beyond that the discussion will be how to get rid of the long-tail v4.”

The gateway techniques required to hook up v6 devices with v4 content will create quality-of-experience problems for v4-only Web sites, Curran says. “If customers are using gateways and translation techniques to get from the living room to your content and you stream and it breaks up, someone is going to call up and say, ‘What’s wrong with your site,’” he notes. “If you’re depending on ad revenues, you’re going to lose advertising information, because the tracking systems won’t know where the customers are.”

Of course, what worries service providers is that any problems subscribers have with accessing Web sites or quality of experience once they do will become their problem. “We have no idea what will happen when we turn on our first v6 subscribers,” Rubinger says.

Clearly, content providers as well as service providers are getting the message that it’s time to move. “For a long time I was a v6 skeptic,” says Sam Gassel, technology fellow for digital media technologies at Turner Broadcasting System. “I thought there were enough v4 addresses available that we could continue to ignore it.”

Last fall Gassel switched gears. “We began to understand the strategies service providers would need to implement to preserve v4 – address translators, carrier-grade NAT, Dual Stack Lite (a Comcast-devised alternative to CGN). That kind of aggregation leads to congestion, packet loss and increased latency.”

Over the past seven months Gassel and his team have come a long way, to where it’s much easier now than it was to implement v6 peering with network companies. “We’re now ready for low-volume tests,” he says.

One cause for concern noted by TBS and many other content providers in early tests of v6 is the fact that some user devices will send out a query to the DNS for connectivity to a Web site via v6, but, then, when the v6 packets are sent won’t recognize that they are v6 enabled and so will reject the transmissions. “We’re only seeing that with 1/2 percent of users, but it’s enough to cause concern,” Gassel says. “That brokenness has been reduced, but we’ve only been doing small-scale tests.”

Even when sites are v6 enabled and devices are working properly in sending out v6 queries and receiving v6 content, there will be issues in the early phases of the transition. While in an all-v6 environment end-to-end communications will run faster than they do in the v4 domain, owing to the nuances in the numbering system that speed packet routing, a v6 packet traversing the Internet today typically goes through a number of tunnels, which can triple the roundtrip time in comparison to v4 packets, according to recent test reports.

Conversely, as ever more sites are optimized for v6, the performance for v4-addressed devices is likely to diminish. “At some point people will optimize their sites for v6 and they won’t be as good at handling v4,” Curran says.

Indeed, notes John Brzozowski, distinguished engineer and chief architect for IPv6 at Comcast, such problems showed up during the World IPv6 Day experiment on June 8, when over 400 sites worldwide, including, prioritized packet handling for v6 packets as a way for stakeholders to see whether the v6 domain works as intended when v6 traffic volume increases. While everything worked well on the v6 side, “the Dual Stack v4 failure rate was higher than expected,” Brzozowski says.

Comcast is working to fix issues it discovered in the v6 Day trial,but much more work along these lines needs to be done, he adds. “We want to see more traffic,” he says. “We need to have more screens for end users, more v6 in their hands.”

Device Issues

Another factor of concern to service providers is the reliability of devices that are promoted as v6-ready at retail outlets. One reassuring note in this regard is sounded by Steven Bosch, executive director of business development strategy at Best Buy. The firm is taking steps to ensure products work as billed and that consumers will be aware of the coming transition to v6.

“Any time we go through a major technology transition like this there’s an increased likelihood that products and services won’t line up,” Bosch says. “When they don’t, the result is an increase in the volume of returns and more confused customers.”

But it’s a work in progress. Asked what percentage of routers on Best Buy shelves are now v6 capable, he responds, “We’re trying to figure that out. Consumer electronics products are our bigger concern. Knowing which ones have been updated, which ones can be updated by firmware and which ones are not upgradable is something we need to know.”

Operators, of course, want to be sure consumers are buying v6-ready products, which means they need to know when everything on retailers’ shelves meets that stipulation. While Bosch is confident that will eventually be the case, he says it’s impossible to say when at this point. “We’re pushing our manufacturing partners to make sure it happens as soon as possible,” he says.

One point of aggravation for operators is the fact that many suppliers of home routers could easily upgrade existing devices to be v6 compliant but are likely to choose not to in favor of requiring consumers to buy new units in order to be v6 ready. “Making people go out and buy new equipment may be a way to generate more revenues, but it’s a disservice to everyone,” says one cable industry executive, asking not to be named.

Apparently, router supplier D-Link doesn’t intend to play that game. “Most of our devices are upgradable,” says Claire Cheng, senior product planner at D-Link. Noting that 23 devices in D-Link’s product line are now v6 certified, she adds, “More than 10 million of our v6-ready routers have been sold into homes and businesses.”

While with the passage of time many issues of concern to operators will become less so, there’s no getting around the fact that, until v6 addresses are allocated and end-to-end Dual Stack is activated with CGN, nobody can know what to expect. But it won’t be long before at least some operators begin discovering what it will take to keep customers happy through the long transition.

“Three months after ARIN runs out of v4 addresses we’ll see CGNs showing up in networks,” predicts CableLabs’ Chris Donley. “This is coming quickly. For some, it could happen in time for Christmas.”