wan.notes

Course notes, terrestrial WAN

Buddenberg/Aug97
 
 

Internet plumbing is made up of LANs, terrestrial (wired) WANs and radio-WANs that are all interconnected with routers. This note addresses the terrestrial WANs.
 
 

WANs differ from LANs in two ways:

The telephone company version of data communications.

Analog and voice. The analog voice history tells you where a lot of the telco digital stuff came from and why there are some rather strange assumptions when viewed from the point of packet switched data.

 The phone company's engineering here is driven by the desire to multiplex multiple telephone conversations onto a single wire between central offices -- saves copper.

[insert analog_switch.gif]

 The human ear can generally hear frequencies from about 30Hz to 30kHz. But passing this much frequency spread through the telephone system was not feasible ... it turns out that if you pass the frequencies in the range 30Hz - 3kHz, the speech is intelligible. Since this is about an order of magnitude more efficient than full fidelity, this what POTS[1] is built on. The voice input is trimmed to the requisite 3kHz (this is what analog filters do), then feed it to a mixer. The mixer is also fed with a fixed frequency from an oscillator. The mix of the two puts places the voice information onto the fixed frequency carrier.

[insert analog_mux.gif]

 If two conversations use different carrier frequencies, they can both use the same wire for transmission to the destination central office[2]. At which point, the process is reversed and the original voice frequencies are piped out the local loop to the end telephone set.

 The analog system had the singular advantage in that it worked. The problems all have to do with distance and scale -- solving them requires amplifiers which amplify both signal and noise. And the more complex the analog network gets, the more opportunities for analog filters (which are by no means leakproof) to generate crosstalk. A digital transmission system eliminates the worst of these problems, and the place where the payoff exists is in the trunks between central offices. Enter the T-carriers as digital transmission pipes ... which connect to the still-analog local loop.

Migrating to digital pipes. The phone companies, being regulated monopolies, upgrade their physical plants based on 1) depreciation schedules and 2) the legacy equipment that they can't replace overnight and don't want to obsolete lest they have to explain to public utility commissions why they've got these white elephants around.

Digitizing physical layer - pipes with exactly two ends. Do not confuse pipes with switches that take data out of one pipe and put it into another. Although the mapping is crude, we're at the physical layer in the ISO Reference Model.

Conveyer belt analogy. A useful way to look at a telco digital pipe is to consider it like a conveyer belt. [insert conveyer.gif]


 
 

Example:

T-1 pipes have 24 buckets per second (64kbits * 24 = 1.54Mb/s)
 
 

Where'd these numbers come from? The Nyquist sampling theorem tells us that if you sample at twice the rate of the maximum frequency, you get all the information. So if we sample at 60kbps, then we capture all the sound in the old analog 3Khz. Which is why all these digital pipes come in multiples of either 56 or 64kbps[3].
 

Local Loop.  There are three technologies for what the phone companies call the local loop -- the reach from the central office to your residence or business.

ISDN pipes have 3 buckets organized as 2 B channels (64kb/s) and 1 D channel which also has 64k (but 48k is used as overhead leaving 16k for user applications). Consider ISDN as moving the analog-to-digital conversion from the central office out to your handset. Now a 3-bucket conveyer belt delivers 64kbit sized buckets of data to the central office where the contents are simply shifted into the same-sized trunk buckets to the next central office.
 
 

All carriers, (T-carriers, ISDN, O-Carriers) run at 8kbuckets/second -- driven by the uncompressed voice assumption -- and have bucket sizes that are multiples of 64kbit. This allows 'channels' to be multiplexed into larger trunk lines. For instance, the two B channels of a residential ISDN connection (BRI, 2B+D) can be mapped onto two of the 24 T-1 channels. (Hint: PRI = 23B+D)
 
 

The big attractiveness to the telcos is that T-carriers (and later OC-carriers) can be swapped in for analog trunk lines on a wholly incremental basis. The shift from analog to digital does not change the circuit switch paradigm one whit.
 
 

Other WAN technologies. Pipes from other than the telcos.

Communications to the home has heretofore been a monopoly of the telephone companies (and the post office). But that is -- at least technologically -- changing.

xDSL. Digital Subscriber Line technology is an alternative to ISDN for local communications. The way it works harks back to the POTS description above. The local loop telephone line is frequency division multiplexed with several 3kHz channels, much the same way analog trunk lines used to be. Each of these analog channels can then be used to haul bits. Commonly, the first one (the 0-3kHz one) is used to carry voice telephone calls -- POTS reborn. The rest have digital signal processors added to each end to provide data transmission and recovery service and a multiplexer that aggregates all these individual sub-pipes together.

By varying the number of send and receive pipes, the number of bits carried in each direction can be toyed with. This is where the variations of x in xDSL came from. ADSL stands for Asymmetric -- an observation that a WWW surfer from home is likely to need only one of these channels to pass URLs upstream; the rest are available to pass pictures of Mars downstream to a browser.

 There are distance limitations to xDSL and the limit is soft -- the bit rate supported decreases as the distance increases. But the advertising hype indicates you should get some reasonable service for runs of around a mile at capacities equivalent to T1.

Telephone empire crumbling. The initial business model was that users would have an xDSL 'modem' at home, use telco local loop to the central office switch, whereupon the bits would join the telco's cable structure. And xDSL service offerings would appear on the telcos rollout schedule.

[insert xDSL_telco.gif]


 
 

But the barriers to entry are low and there's a buck to be made ... an alternative is appearing: Get a 'burglar alarm' circuit that is a hard-wired (telcos call this a 'dry splice') pair that runs between your home modem and your neighborhood ISP's router. All the phone company is providing is the copper. You then buy the xDSL end equipment from an electronics manufacturer out there and connect your home computer to the ISP's router.

[insert xDSL_bypass.gif]

 Note that the standardization requirement is quite limited -- the xDSL equipment must operate correctly at both ends of the wire and the xDSL equipment at the ISP must interfaced with the router. But you don't need world-wide standardization (like ISDN did) to get worldwide access and interoperability over the Internet. Note that the telco's central office switch is simply not part of this infrastructure at all. Not surprisingly, at least some telcos are starting to make the burglar alarm circuits hard to get.

Cable TV. Two thirds of the homes in the United States are visited by the CATV cable plant. The fact that the cable plant is there is fairly well known. And the cable plant itself has both

This by no means solves all the problems, either technically or in terms of business model[4].

There are some interesting electrical engineering problems with internet over cable tv, but most are known and none appear to be showstoppers in the vast majority of cases. A few:

Technically, the Internet over CATV plumbing is looking very much like ethernet today. Which means that the channel between you and the Internet will not be yours alone -- you share it with your neighbors. But the whole-subnet capacity will be in the same neighborhood as 10Mbps ethernet. The tinkertoy assembly is also straightforward -- you'll probably get a box with an RJ45 UTP plug on the side that accepts and produces ethernet frames ... which you can plug into either your computer or a hub.
 

Note also that a CATV 'LAN' doesn't need (indeed, can't use) a switch; its head-end plugs directly into a router. xDSL is similar, although there's a service offering fight in the industry at the moment: an xDSL circuit can go through a telco switch, but it's really an end system-to-router hardwire link so there's no value added by the switch unless you as a user intend to dial in to multiple routers.

 Many of the problems here are rooted in the fact that the CATV industry is wedded to the television industry -- one of the more notorious stovepipes. Most cable companies don't know the difference between internet connectivity and internet services.

Wannabes -- power line carriers. Every two or three years somebody comes up with the bright idea that we can use the power lines that enter homes and businesses to carry data; not a new idea. And several companies have started up and died trying to exploit this market ... never quite seems to take off.

 Why this is a good idea. The wiring already exists in and to your house -- you can hook up a LAN by simply inserting a gadget between your computer's NIC and the 110V receptacle on the wall. Indeed, with a bit of integration, the NIC could be attached to the power cable on your computer so when you plug in the volts, you also plug into the LAN. Technically this is quite feasible for small, low capacity, fairly simple networks.

The practical problem has to do with household wiring and all the other things that hang off it:

None of these difficulties are insoluble, but the cost to do so has not been less than alternatives.
 

Data link layer -- switches -- and the backbone WAN.

Getting through the WAN cloud from your building's router to a destination router is the subject here. How do we connect these various links so they accept and produce datagrams to the appropriate router?

There are about four ways to build the WAN component of an internet.  Note that they're not mutually exclusive: all these technologies have router interfaces.
 

IP over DWDM is becoming popular with large ISPs for several reasons, bypassing the telco's old technology being only one of them.  A practical reason is that the terrestrial WAN becomes simpler: there are fewer boxes with transistors in them between routers.  And since routers -- especially the high end ones that support DWDM -- all have SNMP agents in them, detecting and isolating faults becomes fairly straightforward.  This "Roll-your-own WAN" simply interconnects routers with nail-up pipes (no switching at all). Typically, these routers use SLIP or PPP as the router-to-router protocol[6] .

With those caveats aside, lets look at some WAN switching.  Unless you are a sizable ISP, you probably won't be in the IP-over-SONET or IP-over-glass markets.
 

Telco-provided switching comes in about four types:

The quick hit: Frame Relay is a well understood, well documented and stable technology. It's widely offered by both the RBOCs and several other service providers. Because it's ideal for a router-connected internet, its popular with users. If in doubt, Frame Relay is a very low risk place to start your WAN.
 
 

Historical thread. Like the evolution of transmission pipes, the history of switch evolution is useful too.

Vendor offerings ... not the same as the standards.
 
 

Pacific Bell has Frame Relay switches at various central offices. Their service offering looks like this:

T-1 physical pipe attached to FR switch. Typically you pay for a minimum pipe size (e.g. 128kpbs) and the first 128kbps of offered traffic is tagged with Discard Ineligible flags -- FR switches are not allowed to throw them away when congested. The remainder of the T1 pipe -- the rest of the buckets on the conveyer belt -- is available to you, but any traffic that uses them has the Discard Eligible flag set; switches can throw those frames on the deck when congested. This, of course, is not as big a deal as the telcos think it is because TCP detects and accounts for missing datagrams. These Discard Eligible and Ineligible flags are called Committed Information Rate; PacBell supports CIR; many other RBOCs don't.

 AT&T and MCI support Frame Relay over ISDN; PacBell doesn't.
 
 

Last year's Next Big Thing (tm).

Unmet WAN requirements with existing protocols include: I'd suggest that there are several other requirements, mostly revolving around multicasting, that are also not met. But these are not yet widely recognized (and certainly weren't in the ATM hype). So, we'll live with the above list for this discussion.

 There's a quite plausible story that illustrates the intellectual origin of ATM: The switch folks at Bellcore were watching the transmission people figure out how to multiplex more and more data into the OC pipes. They realized that none of their switching technologies could handle the drinks from those firehoses and needed a new approach.

Ergo, ATM.

Necessary dehype. The 48 byte ATM cell payload isn't a power of two; rather it's a sawing up of the baby -- a compromise between 64 bytes and 32 bytes. Meanwhile 'gigabit routers' have become a practical reality in the Internet core. If we can do that, why do we need all these messy switches out in the WAN cloud anyway?

D. I used to think that ATM was probably doomed to success due bandwagon effect[9]. Technical shortcomings beaten into submission with brute force:

E. Architectural note: ATM represents a different paradigm. Cell switching requires homogeneaity at the data link layer. Three Dirty Little Secrets about ATM ATM, while appropriate technology for the applications it was originally designed for, has three severe architectural shortcomings: in availability, multicasting and media access. All three of these requirements are critical to military applications.

Availability and survivability. One fundamental characteristic of router-based internetworks is that broken routers or links can be routed around (assuming rich enough connectivity). If a router fails, its adjacent routers will simply not use the links to tht router, and the end-system TCP implementations will service for any packets that were lost with the failed router. Similarly, in local area networks, an FDDI DAS LAN will wrap on failure of a node reconstituting the network. All this is transparent to applications. This is a primary characteristic of connectionless IP networks and TCP state machines implemented on participating end systems.

 ATM, conversely, is a connection oriented protocol. All cells associated with a particular session take the same route through the switching fabric. This means that a switch failure along this route during a session results in a failed session. This is anything but transparent to the application -- it must re-establish the session.

 Why is this important in military systems? The designers of the ARPANET consciously created a basic design that is highly survivable, and automatically uses all the redundant connectivity available to it. They were right.

 Switch vendors say that ATM SVCs will transparently re-establish, but PVCs definitely will not.
 
 

Multicasting. In real world networks, a sensor always sends data to more than one decision support node and decision support nodes are fed by multiple sensors -- a many-to-many relationship. You can easily support this with two arguments:

ATM simply doesn't multicast. In theory, the standard is being patched to fix this. On the other hand, LAN protocols inherently multicast (an ethernet frame dumped onto the LAN is 'seen' by all the other nodes on that LAN). Routers running Multicast IP also multicast by replicating datagrams as required to efficiently reach all necessary parties.
 
 

Media access. This shortcoming becomes evident in radio-WANs where the shared media requirements attend an implementation of any scale. Because multiple radios share the same media, a means of disciplining them is required so they don't all transmit at once. ATM, like all WAN protocols, lacks such a media access protocol entirely.

Telephone company networks consist of point-to-point links connecting switches that switch calls, cells, or frames. Consequently, there is no need for media access protocols, and indeed, they do not exist in POTS, X.25, Frame Relay ... or ATM[10].
 
 

While there are no particular objections to ATM switching in terrestrial WANs, DoD programs should be wary of transposing ATM terrestrial technology into radio-WANs. The 'prove it' checklist should definitely include these points.
 
 

Precise language.

As a final thought, we may use some precision of language to sort out discussions of pipes and switches.

Speed. It is appropriate to use the term 'speed' in guaging the capability of a swtich (or router or hub). As in CPU speed, number of instructions processed per second, number of packets processed per second, ... It is not appropriate to use the term with reference to transmission.

Capacity. Transmission paths, whether copper, fiber, or ether, are properly guaged using the term capacity. A T1 pipe has a capacity of 1.5Mbps; an FDDI LAN has a physical layer capacity of 100Mbps, etc. But the bits go down the wire at exactly the same speed (in copper and fiber, about 2/3 the speed of light) in both of these pipes. Indeed, so do the bits travelling down the wire from your 14.4 kbps modem. Even an OC24 pipe with a 4Gbps capacity still has the bits travelling at about the same speed. The difference is that in the OC24 pipe, there are a whole lot more of them multiplexed into the same conveyer belt.

[1]Lingo for Plain Old Telephone Service

[2]Analog radio works the same way, except there's a second reason to move the frequency up. In addition to placing the information onto it's own frequency, the carrier mixing produces a frequency that can actually be transmitted over the ether -- audio frequencies are too low.

[3]There's some arcane bitstuffing explanations behind the differences between 56k and 64k -- four our purposes, this is rounding error. And all of these sizing decisions were locked into concrete before any voice compression technology leaked into the telcos' life.

[4]One cynical observation I've heard is that CATV technicians were the ones too dumb to get telco jobs.

[5]Although the reverse seems to be true in the local dealings between the City of Monterey and TCI

[6]Router-router interconnect was the original motivation for SLIP and later PPP. The dial-in from home use for PPP came later.

[7]See Malamud, Stacks: Interoperability in Today's Computer Networks Prentice-Hall: 1992 for good discussion. This problem is a critical protocol problem in radio-based networks too.

[8]The tree, not the company that adopted the name.

[9]This comment made it into my notes before the hype wore off. ATM was touted as technology that would cure baldness for the first half of the 90s. Reality checked in about 1995 and ATM is now recognized as a core WAN switching technology -- what it was originally conceived for -- and of limited use beyond that. Naturally, the word takes time to get out ... parts of the DoD were really seduced by the hype and haven't yet recovered.

[10]It is fair to note that this is a shortcoming of all wide area network protocols, not just ATM.