F E A T U R E S    Issue 2.10 - October 1996

Netheads vs Bellheads

By Steve G. Steinberg

The most vicious battle on the Net today is a secret war between techies. At stake is nothing less than the future of cyberspace.



It was a frequent observation among the laptop-toting 25-year-olds who crowded into the University of California at San Diego auditorium on an overcast morning last February that if a bomb were to go off right then, the entire Internet would collapse. It was the kind of braggadocio you hear among any large gathering of engineers, but, in this case, it was probably true.

The 250 engineers who filled the dark, wood-panelled auditorium during the two-day meeting of NANOG, the North American Network Operators' Group, were from America's largest Internet service providers - companies like UUNet, Netcom, and Sprint - and they possessed the self-confidence that comes from operating millions of dollars in bleeding-edge technology that the world increasingly depends on. They were the builders of a new age.

The first technical presentation that morning was by Warren Williams, a genial, chubby guy in charge of the Pacific Bell network access point, or NAP. NAPs are a lot like motorway interchanges - they allow traffic to flow between the independent networks that make up the Internet. Williams's big news was that PacBell was about to upgrade its NAP with higher-speed equipment that uses ATM - or asynchronous transfer mode - technology. By taking advantage of ATM, he explained, networks will soon be able to exchange traffic at speeds of up to 660 Mbps, instead of the current maximum of 45 Mbps. While most of the audience listened with a fair amount of interest, laughter kept erupting from a small cluster behind me.

That dissident cluster, I learned from the person sitting next to me, included some of the most important engineers in the room: people like John Curran, the chief technical officer for BBNPlanet; Yakov Rekhter, a lead engineer at Cisco; and Sean Doran, the senior network engineer at SprintLink. These people, the engineer seated next to me said, believe ATM is a flawed technology, one that causes more problems than it solves. They see PacBell's interest in it as further proof that the regional telephone companies are doomed to be incompetent bumblers whenever they move away from their beloved voice networks.

For years, the conventional wisdom has been that ATM will be the saviour of the Internet - ending bandwidth scarcity and bringing about a new era of network reliability. The technology is routinely described by the computer trade press as revolutionary and inevitable. It has the backing of everyone from AT&T to Sun. Yet, I now discovered, some of world's smartest and most powerful Internet engineers find the technology laughable.

This discrepancy, it turns out, is not just a minor anomaly, or the result of a few opinionated extremists. Instead, it is a critical battle in a war between two fundamentally opposed groups of engineers - a war between the Bellheads and the Netheads. In broad strokes, Bellheads are the original telephone people - the engineers and managers who grew up under the watchful eye of the old Bell telephone monopoly - "Ma Bell". They believe in solving problems with dependable hardware techniques, and in rigorous quality control - ideals that form the basis of the robust US phone system and that are incorporated in the ATM protocol.

Opposed to the Bellheads are the Netheads, the young Turks who connected the world's computers to form the Internet. These engineers see the telecom industry as one more relic that will be overturned by the march of digital computing. The Netheads believe in intelligent software rather than brute-force hardware, in flexible and adaptive routing instead of fixed traffic control. It is these ideals, after all, that have allowed the Internet to grow so quickly, and that are incorporated into IP - the Internet Protocol.

The two protocols embody very different visions of communications, leading to connected worlds with different social patterns, commerce and even politics. In extreme terms, think of the difference between the chaotic world of the Web and the rigorously controlled, financially lucrative world of 0800 numbers.

Today the defenders of these two visions are jockeying for jobs with power over networking infrastructure, manoeuvering for control of the switches, and struggling to implement or thwart ATM. If the two sides agree on anything it is this: as ATM goes, so goes cyberspace.

The man who runs the Net

Nobody runs the Internet, or so they say. True up to a point. Now meet Sean Doran. We learn from his Web page that he is 26 years old and a Canadian citizen. He is, in his own words, "a person who is deliberately cheeky toward people in authority, rude to people who think that they deserve respect that they have not earned, and who generally causes trouble on IRC." He is, in every way, completely different from what you might expect of the person responsible for the technical administration at the heart of the Internet.

Doran is lead engineer at SprintLink, a small guerilla operation within US telecom giant Sprint. In person, his manner is rather high-strung - always fidgeting, interrupting and blurting out great huge paragraphs. He's extremely articulate; every question elicits a ten-minute response, which, you get the feeling, is done more for some abstract notion of completeness than for your edification. When challenged on a point, he tilts his head, and his eyes begin to blink faster as he overwhelms your argument with masses of examples, quotes, and anecdotes. There is no better way to evoke this response than to try to argue in favour of ATM.

"How do you scare a Bellhead?" he begins. "First, show them something like RealAudio or IPhone." Then tell them that right now performance is bandwidth-limited, but that additional infrastructure is being deployed. Then "once they realise their precious voice is becoming just a simple application on data networks," point out that "the Internet is also witnessing explosive growth combined with 45% returns on investment, compared to 5% growth and only 12% or so returns for voice." In short, says Doran, make Bellheads realise they are witnessing their extinction.

"How does a Bellhead react?" he continues. "Usually a 'hmm' or even a polite 'that's neat,' followed by desperate attempts to kill the Internet in any way possible. One way is to try to take control of it by controlling the transmissions layer - by trying to make ATM supplant the packet-switching fabric."

I point out that Vint Cerf, "the godfather of the Internet," says the opposite: he claims there is no conflict between the telecom and data divisions at MCI. "We don't have camps," Cerf had told me. "Our voice engineers work hand in hand with our data engineers." To that, Doran quickly retorts: "Vint Cerf is on drugs. He should start figuring out why so many MCI engineering résumés are circulating."

Who Foots The Bill?

At the heart of the ATM debate is really an older argument," says Brian Reid, the elder statesman and gnomic director of Digital's networking systems laboratory. "It is the debate between packet-switching fans and circuit-switching fans; two sides with irreconcilably different points of view."

Circuit switching goes back to the original days of the telephone network, when switchboard operators would connect two phone lines with patch cords to create an electric circuit. Today the circuits are virtual rather than physically distinct wires, but the principle of setting up - of predetermining - a connection remains.

When I use the phone to call a friend at MIT from my office in San Francisco, the telephone switches between us first decide on a route for our voices to travel. The route might go from San Francisco to Austin to Newark to Boston. Once that connection is established, my friend and I have full use of that conduit for as long as the connection stays up. But if one of the intermediate switches is destroyed by some calamity, say, in Austin, then our connection dies.

The Internet, on the other hand, doesn't require connections to be set up ahead of time. I can just send a packet to my friend without waiting for permission. How the packet gets there depends on the second-to-second status of the network's topography. Internet packets act like tiny autonomous agents that are able to find their own way. Both the advantages and disadvantages of packet switching are due to the difficulty of keeping track of traffic that can come from any direction. It makes packet-switched networks harder to destroy or censor, but it also makes them harder to manage.

"Traffic management" is a phrase with almost totemic weight among telecom engineers. So it's not surprising that when they started looking at data traffic in the 1970s, their first thought was to make the packet switching of the Netheads act more like the circuit switching traditionally used for voice. Or as David Sincoskie, one of key people who helped make ATM a reality, tells it: "Our goal was to prove we could achieve the same quality of service as circuit switching, but with packets." Which is exactly what ATM does. Voice and data are both split into small, equal-sized packets. But a virtual connection, called the VC, must be established before these packets can be sent. That's because ATM packets are lobotomised: instead of knowing where they want to go, as in a true packet-switching network, they just know the ID number of the virtual circuit they belong to, and they must stick to that predetermined path. Even though hundreds of different connections end up multiplexed on a single fibre optic pipe, every ATM connection is treated as if it has its own, narrower pipe.

There are some technical arguments in favour of ATM's approach. For example, it takes fewer bits for a packet to carry a virtual-circuit ID number than it does to carry a complete destination address. And since the table of virtual-circuit IDs is smaller than the complete table of destination addresses, ATM switches can do the necessary lookups faster.

But the decisive argument, at least from the telecom point of view, is that it is easier to manage traffic when it's cleanly separated into unique connections. Billing makes for a good - and not unimportant - example. It's very hard to count IP packets and decide who should pay for them. But it's easy to keep track of who opens a connection and how long that connection stays open. In short, ATM would allow Internet providers to charge by usage, instead of flat-rate.

That's an ability many experts, even outside the Bellhead coterie, are quick to embrace. Economists like Hal Varian of University of California at Berkeley have long argued that some kind of usage-sensitive charging system must be instituted for the Internet to remain sustainable. Flat-rate pricing, after all, doesn't provide any economic incentive to refrain from wasting the Net's limited bandwidth. One result is undergrads who, for $29.95 a month, clog up the Internet with CU-SeeMe sessions. But these arguments don't fly at all with Nethead zealots. As PARC's Steve Deering delights in pointing out: "IP is hard to charge for? That's not a bug, that's a feature!"

Playing in the traffic

Generally, Netheads' dislike of ATM's circuit switching in drag has to do with how they think about traffic. "Essentially, ATM is a very predictable technology. There is no scope for anything that doesn't fit the model", says Doran in full-power didactic mode.

Bellheads like to make traffic predictions, and design systems accordingly. Netheads cheerfully admit they have no idea what traffic will look like next month. It's easier, they say, to have the packets fight it out among themselves than to try to force some kind of order. Traffic is unpredictable, spontaneous, liable to jump up in strange patterns. Doran sounds almost mystical when describing the traffic that flows across his network: "Each time one thinks one understands the variables, more appear." This philosophical divide should sound familiar. It's the difference between 19th-century scientists who believed in a clockwork universe, mechanistic and predictable, and those contemporary scientists who see everything as an ecosystem, riddled with complexity. It's the difference between central and distributed control, between mainframes and workstations.

Only recently, however, have Netheads been able to point to analytical evidence, rather than just anecdotes and vague impressions, to support their worldview. In 1993, a group of mathematicians at Bellcore and Boston University published a paper that showed that data traffic and voice traffic behaved differently. In data networks traffic remains "bursty" at every conceivable scale, while in voice networks it evens out. That makes it hard to optimise data networks, because if you provide only enough capacity for the "average load," then you'll get swamped by the inevitable bursts.

The piece of the ATM protocol most likely to blow up when faced with massive bursts of traffic is what the Bellheads call quality of service (QoS), and it represents the second of the three fronts in the war between the telecom and data communities. It's the debate over whether to make hard guarantees about the quality of service customers can expect, or whether to promise only "best effort." Malthusians versus Cornucopians

Jim Diestel, the director of Internet NAP marketing at PacBell, walked me through the reasons why ATM is the way of the future. He spoke of how, because ATM had been designed with gigabit speeds in mind, it was much easier to build fast ATM switches than fast IP ones. Diestel explained that, thanks to ATM's status as an international standard, more and more manufacturers were entering the market, driving prices down to unprecedented levels. And Diestel twice mentioned how ATM technology would bring a higher quality of service to the Internet.

What he didn't mention was that PacBell had tried to use ATM equipment in its NAP once before, back in 1994. That attempt ended in disaster: service quality plummeted and the equipment had to be thrown away. The paradoxical reason: ATM's vaunted ability to provide high-quality service.

An IP switch handles packets on a first-come, first-served basis. That's nice and simple, but it can lead to congestion problems. ATM , on the other hand, tries to ensure that everyone gets an equal share of bandwidth by specifying how often each virtual circuit can send a packet. In order for this to work, the ATM switch must store all the untimely packets in memory until they can be sent. And there will be a lot to store. Because data traffic comes in uneven bursts it will never follow the steady distribution that ATM desires. This is the unfortunate reality that killed PacBell's original ATM equipment: untimely packets filled up the switch's memory, forcing it to drop additional packets on the floor and hope no one would notice. But people did, and PacBell's new switch has a whopping 192 megabytes of memory on board just to buffer packets. This memory ends up being the single most expensive part of the switch - sometimes as much as 30% of the total cost. Why not get rid of it?

Instead, Netheads say, forget about quality-of-service guarantees. Bandwidth is cheap - simply overengineer the network to provide so much of it that nobody cares about fairness. If the lines are running at 100 Gbps, as they should be in a few years, nobody is going to care if their packet gets stuck behind a bunch from someone else - it will still be plenty fast. Plus, say the Netheads, Internet applications should adapt to traffic conditions, not demand a fixed amount of bandwidth. Quality should be a variable - absolute anathema to Bellheads.

Plenty of technical arguments are made in favour of quality-of-service guarantees. They all come down to one basic confrontation: Malthusians versus Cornucopians, with the Bellheads predicting bandwidth scarcity and the Netheads predicting abundance.

Big packets

The third and final division between the Bellheads and the Netheads is over something terribly mundane: the number of bytes in an ATM packet. Yet this may be the fault line that ends up mattering most. Packet size is the primary reason Sean Doran hasn't deployed ATM equipment in Sprint's network; it is also the reason why MCI's Vint Cerf did.

An IP packet has a 20-byte header and a variable-length payload of anywhere from 12 to 65,615 bytes. Contrast this with an ATM packet, technically known as a cell, which is 53 bytes long. The first five bytes (called the header) contain information about the cell and its connection; the next 48 bytes (the payload) contain the actual data. It's difficult to convey how insane ATM's cell scheme sounds to anyone in the data community, but it's roughly equivalent to Ford announcing a new car shaped like an upright obelisk. Sure, it could be made to work, but it's neither aerodynamic nor practical. If anyone did something that dumb, you'd suspect political meddling. You'd be right.

In 1988, International Telecommunications Union delegates met in Geneva to decide on the length of an ATM packet. It was immediately obvious that consensus was going to be difficult. The Europeans wanted 32-byte payloads, because that would be best for voice, while the Americans and Japanese wanted 128-byte payloads, since that would be better for data.

The discussion quickly turned confrontational, with the US and France becoming the main combatants. Pressure built to solve the question the diplomatic way: split the difference. And so was born the 48-byte payload, which, combined with a 5-byte header (the smallest that the ITU could agree upon), added up to a 53-byte cell.

No one was terribly pleased with the result. Sandy Fraser, one of the men who developed the core ATM techniques at Bell Labs in the 1970s, was forced to watch his technology haggled over like some kind of border dispute and now says, "In my view, they picked the worst of both worlds." The Netheads were far more vocal. T-shirts and buttons with a slash through a red 53 became popular at Internet conferences. Sean Doran practically sputters when the topic comes up, claiming that the short cell size incurs a 30% overhead.

Netheads like standards to evolve. Bellheads like them fixed. That way, manufacturers face less risk: they don't need to worry that their products will quickly be rendered worthless by a competing standard. We are currently watching these two approaches play out in the switch and router industry. While datacentric IP router manufacturers have hung back, waiting to see where the market is headed, ATM switchmakers have leapt ahead, secure in the knowledge that their standard is backed by the world's mammoth telecom companies. Hence, it is possible to go out and buy an ATM switch today that runs at 622 Mbps. But top-of-the-line IP routers from Cisco and NetStar still go up to only 155 Mbps.

This is why some ISPs, including MCI and @Home, now use ATM in their networks. "We were forced to embrace ATM," explains Rob Hagens, MCI's director of Internet engineering. "We wish we didn't have to, but the only equipment out there that can handle the speeds we need is ATM-based."

Although MCI uses ATM in name, the company certainly doesn't use it in spirit. It has, as much as is possible, lobotomised the technology. Instead of having ATM constantly opening and closing new connections each time a user links to a Web site, MCI leaves a single wide connection "nailed up" between the main hubs along its backbone. And MCI currently turns off ATM's attempt to guarantee quality of service, since it breaks down in the face of unpredictable traffic.

Truce

At an anonymous office park in Palo Alto, California, a small start-up called Ipsilon is building the foundation for what will come once the war between the Bellheads and Netheads ends. Ipsilon was founded in 1994 by Tom Lyon, who had been involved with Sun's initial ATM efforts. Big and shy, prone to staring down at the floor as he speaks, he is now Ipsilon's chief technical officer. A number of other well-known data networking people soon joined the firm, lured by the opportunity of doing something so innovative that it might allow tiny Ipsilon to challenge Cisco's dominance.

Bob Hinden, probably best known for his work on IP version 6, became director of routing software. Dennis Ferguson, a longtime Internet backbone guru, joined as a member of the technical staff. Brian Nesmith, a former vice president at Newbridge Networks Corp's Vivid Group, became Ipsilon's CEO. All of them were data guys who had lost their Nethead religion - or at least learned to soften it with a dose of pragmatism. In early 1996 the company was 50 employees strong, and had two rounds of venture capital financing behind it and a prototype IP switch up and running.

The IP switch is Ipsilon's flagship product and a paean to cost-efficiency. The switch is built from off-the-shelf and salvaged parts, and obsessively engineered for speed. While other switches are powered by custom or top-of-the-line microprocessors, Ipsilon uses a $500 Pentium Pro. And while Ipsilon uses ATM, the company ignores many of the complexities that make the technology expensive. ATM's ability to guarantee different kinds of QoS is thrown out, as are many of the higher-level protocols the ATM Forum, an independent standards body, has secreted like a thick crust over the last five years. Ipsilon uses ATM to quickly switch bytes between different input lines and output lines - and that's it.

According to Lyon, the decision to use ATM was purely pragmatic. "In an idealistic world, I wouldn't go near ATM," he says. "And in theory, pure IP could be as cheap as ATM. But currently the chips necessary to implement ATM are available at a lower price." And Ipsilon can use ATM without falling victim to many of its flaws. The end result? "A configured Cisco runs $75K to $130K. We cost $46K to $50K," says Lyon.

So will ATM die out? Or will it live on inside IP switches? Lyon's answer is subsumption rather than extinction. The result will be a sort of technological sedimentation: our networks will be built on strata of technology from earlier eras. At the lowest level of next-generation networks will be the telecom community's ATM. Built on top of that layer will be the data community's IP, and above that may be the Internet's RSVP, a scheme being developed by a group of IP advocates which provides the main benefits of ATM's QoS mechanisms but which is far more adaptable to different traffic patterns - a software solution to what ATM does with hardware.

Data networks will subsume voice networks, but the ghosts of telecom will live on in the underlying, invisible technology.

In this sense, neither the Netheads nor the Bellheads will have "won" the war. While the Netheads may come out on top, their success will rest on the technical foundations laid by the Bellheads. Each side ultimately will see its ideals incorporated into the network. At the lowest level of networking, the hardware approach favoured by the Bellheads makes sense. Higher up in the protocol stack, the Netheads' software approach is better.

As the inevitability of this result becomes increasingly apparent, there are signs that the Netheads and Bellheads are ready to call a truce. Religious fervour has a way of dissipating when confronted with large amounts of cash; Netheads and Bellheads are starting to realise that if they put aside their differences and concentrate on building better networks with whatever tools are best for the job, they will be richly rewarded.

Even Sean Doran will admit that as enjoyable as it may be to poke fun at Bellheads, they still have plenty to contribute. "There is a useful tension between the two sides," he concedes. It's an admission made easier by the fact that, according to Doran, it's largely the telecom guys who are giving ground. "We had a meeting recently where these very senior executives from Sprint came to our building completely dressed down - trying to fit in. They actually made fun of the one guy who did wear a suit."

"Everyone is learning to work together," Doran continues with a touch of satisfaction, "so we can make lots of money and play with our big toys."