Communications

Pouzin Society - organizational meeting, today

The most interesting data networking ideas I've come across in the past N years are in the recent book by John Day, Patterns in Network Architecture: A Return to Fundamentals.  John Day's fundamental insight is that data networking is a distributed version of inter-process communications (IPC).  Those of us who come at this from the computer and computer operating system side of the world are quite familiar with IPC within one computer running one operating system.

The Pouzin Society (named after datagram inventor Louis Pouzin) is a group interested in investigating the implications of John Day's architecture and how it might be used in developing viable solutions to the current Internet architecture crisis.

Here's a top level picture (from a talk by John Day).  If you don't buy the book, there is an interesting summary in a presentation here.

Viewing data networking as a distributed IPC function (DIF) does a number of things.

• It clears up names versus addresses in a fashion that makes sense to me.  The application asks for connection to another named application while node addressing remains internal to the DIF.  Indeed, the point of attachment address for one DIF is a "name" presented to a lower level DIF.
• It makes management explicit.  To even establish an IPC, we must ask the DIF for the other party by name and then get their permission to exchange information (i.e. to establish the IPC session).
• It simplifies layers.  There is only one kind of layer, the DIF, that is used recursively.  Successive levels in the recursion differ in scope, and likely in policies, but architecturally they are the same.

I'm still getting up to speed, but this appears to be a fundamental theoretical breakthrough.  There are a mix of academics and practical networkers in the room for the Pouzin Society kickoff, so hopefully some practical implications for the evolution of the Internet will emerge in time.

Google's Peering and Caching Strategy

This is a followup on yesterday's post about how little Google/YouTube pays for bandwidth. Google wants to promote peering with ISPs, so they give presentations at ISP meetings.  After reading yesterday's post, Alex Benik of Battery Ventures sent me a link to this presentation given by Google at a 2008 meeting of the Latin America and Carribean Internet Addresses Registry (LACNIC).

As expected, Google peers with as many relevant ISPs as possible.  For the ISP, peering with Google eliminates their upstream costs for traffic to Google.  Since Google represents a substantial volume of traffic for most ISPs, this is a big saving.   As of May 2008, Google was present in 33 public Internet exchanges around the globe, so major ISPs already have connections in places where they can peer with Google.  The minimum qualifications are 5 Mbps of Google traffic and the ability to interconnect using Gigabit Ethernet at one of these 33 major Internet exchange points.

Google Global Cache

What's interesting is Google's caching strategy.  Just as Akamai puts servers in ISP's local facilities, Google is providing a distributed cache for their content.  This available to larger ISPs and allows them to serve Google content directly at the edge of their networks, thus reducing traffic on the ISP's backbone network.  Here's a representative rack that Google provides to the ISP.

YouTube's fine - Analysts don't understand Internet peering

As widely reported, Credit Suisse analysts have estimated Google's YouTube may lose $470M in 2009 and more in the future.  However, their estimates say Google will pay $360M for bandwidth in 2009.  I don't know how Google figures their cost of bandwidth, but anyone who understands anything about Internet transit/peering knows Credit is way off base.

Google does not pay for Internet Transit the way most tier 2/3 ISPs or most content providers must.  The economics are simple.  If you are a Tier 2 ISP, you have to purchase Internet Transit services from a Tier 1 network to handle that customer traffic which goes off your network and for which you cannot make other arrangements.  The most notable 'other arrangement' is peering.  If you have significant traffic to/from another specific network, you and the other network can both save Internet Transit costs by exchanging traffic locally, i.e. peering.  Of course an enormous amount of your traffic is directed to Google.  If you have a presence in any data center where Google has a presence, you would love to peer with Google, as that saves an enormous amount on your payments for upstream Internet Transit.

A similar effect plays out among Tier 1 providers.  If one tier 1 network cuts a special deal with Google, Google routes all their traffic through this provider and suddenly the other tier 1 networks have large asymmetries in their tier 1 peering arrangements.  Either they also cut deals with Google or they have to renegotiate their tier 1 peering arrangements to pay for the traffic asymmetry (something that's highly unlikely!).  Google is the one with leverage here!

I don't know what, if anything, Google pays for bandwidth, but it's not paying $360M for Internet transit.  Sorry Credit Suisse, you better go back to analyzing derivatives, credit swaps and other purely financial plays.

Google does have costs.  They have data centers in many parts of the world and they have a private fiber backbone that interconnects their sites and connects their private network to many, many potential peering points.  Operating their private backbone is a real cost to them and I haven't examined their financial reports to see if there is any way (from public data) of estimating their costs for this private network.

But until someone does this analysis, forget what you've read from Credit Suisse.

40th anniversary of RFC 1

April 7, 1969 was the day a graduate student at UCLA, Steve Crocker, published RFC 1 to the somewhat informal Network Working Group associated with the ARPA Network project.  I have heard Steve Crocker speaking informally about those days and my understanding is he created what became the "Request for Comments" series because anything more formal would have brought down several layers of governement bureaucracy.

It's interesting that the Internet owes its origins to a government project, but a lot of what happened was done by individuals and groups working around and/or in spite of government processes.

More bandwidth ― less delay, less latency

I was a little sloppy yesterday and several people have questioned my comment about latency.  I was reacting to slide by Herman Wagter of Amsterdam's Citynet in his presentation at F2C 2009.  His slide said:

Latency is the cause, bandwidth is the effect.

From his discussion it was clear he meant "Latency is the cause, bandwidth is the cure."  At the time he was talking about real-time person-to-person communications and illustrating it with a housebound person in Amsterdam who wanted to play cards with friends in other places (not nearby).  Verbally he mentioned the issue of sending large files.  In short he was addressing the real reason for high capacity Internet access links.

Why people want more "bandwidth"

It's not because they need or want to send and receive 100 Mbps of data all the time or even a significant part of the time.  The issue is delay, specifically serialization delay.  If I have a 1 Mbps upstream Internet connection and I want to send an email with a 5 MB Powerpoint file attached, it will take more than 40 seconds (5 MB ~= 40 Mbits).  On a 100 Mbps link, the same email is delayed only a fraction of a second.

Serialization delay also effects media streams, although much less.  If I want to send 500 Kbps of continuous video over that 1 Mbps uplink, serialization delay will cause added latency.  IP is a packet protocol and the 500 Kbps video stream will be broken into a stream of packets, typically ~1500 bytes (12 Kbits) each. While the serialization delay is only 12 ms on the first link, there is serialization delay on every link.  If there is another 1 mbps link at the other end, that's another 12 ms of delay.  And here, 12 ms is significant.  For a natural interaction between two people, you'd like to keep the round trip delay below 200 ms.  Nothing goes faster than the speed of light so transoceanic links introduce many 10s of ms of delay, each way.  It is very easy to eat up a 200 ms budget, so saving 12 ms at each end is significant.

Burst rate versus continuous

For me and for most people, the issue that drives demand for high speed access links is delay, not the amount of information that is to be sent or received.  Indeed, I'd love a service offered only a few Mbps average over a month, if I could be guaranteed 1 Gbps on a burst rate basis whenever I wanted.

David Weinberger is blogging F2C rather completely

And David is a expert blogger!  i.e., really good coverage.
  http://www.hyperorg.com/blogger/

The first Freedom to Connect conference entry is here. 

Latency is the cause; bandwidth is the solution

Actually, what Herman Wagter of Amsterdam's Citynet said at F2C 2009 was:  latency is the cause, bandwidth is the effect.  But his explanation matched my title above.

If you are attempting to interact with other people, whether by VoIP or just playing cards together (with video) you need less than 200 milliseconds of end-to-end delay.  If it's playing cards together, with video, and you need to exchange 500 Kbps in less than 200 ms, you need a 100 Mbps pipe!

It's latency that drives the need for high bandwidth.  Most people won't fill that pipe most of the time, but they need the pipe to guarentee that what they do send gets through rapidly.

Highlights from Day 1 of F2C

Shamelessly ranked by my areas of interest...

Tim Nulty of East Central Vermont Community Fiber.  Tim is a veteran network builder and a forceful speaker, so he's happy to tell it like is.  He's also got that Yankee mix of liberal politics with extreme fiscal conservatism.  He's building fiber networks, in rural Vermont, which pay for themselves.

Ken Biba of Novarum has been measuring actual wireless networks in buildings and in cities for years.  While the detail is in a report available for purchase, the summary is that WiFi-based Muni WiFi yields significantly better performance than 3G cellular.  Interestingly coverage and reliability is right up there in selected cities, as well.  The take-away - 802.11n really rocks. I.e., the next cycle of WiFi is going to be vastly better than what he's been measuring over the past 3 years.

Ellen Miller of Sunlight Foundation was low key by comparison with Tim or Ken, but her stories were compelling - multiple instances of Internet community feedback creating the kind of information that the "open government" initiatives aspire to.

Finally, Dewayne Hendricks is always interesting.  This year he seemed more optimistic than last, presumably the result of the recent election.  In any event, here's another speaker with deep experience in building networks.

Freedom to Connect 2009 and Municipal Fiber

I'm attending F2C 2009 in Silver Spring Maryland. If you are here, please say hello.

Things are just getting started with a panel on Municipal Networks, led by Joanne Hovis, President of Columbia Telecommunications Corporation (CTC).  Panelists are:

Tim Nulty, formerly of Burlington Telecom and now running East Central Vermont Community Fiber, has tons of experience building fiber networks in low density areas (Vermont).

Dirk van der Woude, from Amsterdam's CitiNet, to talk about the Amsterdam's municipal fiber to the home project..

Lev Gonick, CIO Case Western Reserve, who was a key player in creating a 4000 mile fiber network for Cleveland and northeast Ohio under a community organization called OneCommunity.

Bill Schrier, CTO for Seattle, which is starting a fiber project, but already has it's own electric power utility.  (Although Bill implies they have had to drag their utility brethren into this).

What's interesting is the discussion on the chat backchannel is not about muni vs. commercial, but wireless versus fiber.

Tim Nulty has a strong argument that wireless is excellent for mobility, but not economical for fixed access.  In rural Vermont, a WiMAX network would cost $35M if you could get access to the spectrum (which is being horded by others).  Fiber would cost $70M but has 50 times the capacity and several times the revenue potential versus the wireless approach.  Further, if you deploy wireless as an addon after you have the fiber network (and the customer support infrastructure), the incremental cost is dramitcally less (perhaps $10M) and you get enough incremental revenue to get a good return on investment.  IN other words, you make more money if you do fiber first.  Tim's key to success is to get as near complete deployment as possible - something that is possible in areas where the incumbents are going.  Second, he goes for community ventures as a way to qualify for muncipal bonds.

Shareholders should demand phone companies open their fiber networks

A few weeks ago, Benoît Felten of Yankee Group and Fiberevolution gave a speech at a New Zealand Commerce Commission event which included a fascinating argument.

He points out that receiving a return on the substantial capital investments that a fiber to the home (FTTH) project requires is much more dependent on takeup rates than on the average revenue per user.  

Single player large scale deployments usually achive only 20%-25% initial adoption after which growth is exceedly slow.  On the other hand, systems which are open to competitors, i.e. with viable wholesale services, attract many more players who market, sell and deliver new services thus dramatically increasing adoption and accelerating the wholesaler's return on investment.

In short, obtaining a regulatory holiday so you can run your new fiber network as a monopoly is actually bad for your shareholders!

Benoît's speech was filmed.  For the full presentation (in four segments of < 10 minutes each on YouTube) see Benoît's post.

If you want just the essence, listen to the first three minutes of this:

Here is an enlarged version of the chart that Benoît is using while he speaks: