Communications

Google's Peering and Caching Strategy

This is a followup on yesterday's post about how little Google/YouTube pays for bandwidth. Google wants to promote peering with ISPs, so they give presentations at ISP meetings.  After reading yesterday's post, Alex Benik of Battery Ventures sent me a link to this presentation given by Google at a 2008 meeting of the Latin America and Carribean Internet Addresses Registry (LACNIC).

As expected, Google peers with as many relevant ISPs as possible.  For the ISP, peering with Google eliminates their upstream costs for traffic to Google.  Since Google represents a substantial volume of traffic for most ISPs, this is a big saving.   As of May 2008, Google was present in 33 public Internet exchanges around the globe, so major ISPs already have connections in places where they can peer with Google.  The minimum qualifications are 5 Mbps of Google traffic and the ability to interconnect using Gigabit Ethernet at one of these 33 major Internet exchange points.

Google Global Cache

What's interesting is Google's caching strategy.  Just as Akamai puts servers in ISP's local facilities, Google is providing a distributed cache for their content.  This available to larger ISPs and allows them to serve Google content directly at the edge of their networks, thus reducing traffic on the ISP's backbone network.  Here's a representative rack that Google provides to the ISP.

YouTube's fine - Analysts don't understand Internet peering

As widely reported, Credit Suisse analysts have estimated Google's YouTube may lose $470M in 2009 and more in the future.  However, their estimates say Google will pay $360M for bandwidth in 2009.  I don't know how Google figures their cost of bandwidth, but anyone who understands anything about Internet transit/peering knows Credit is way off base.

Google does not pay for Internet Transit the way most tier 2/3 ISPs or most content providers must.  The economics are simple.  If you are a Tier 2 ISP, you have to purchase Internet Transit services from a Tier 1 network to handle that customer traffic which goes off your network and for which you cannot make other arrangements.  The most notable 'other arrangement' is peering.  If you have significant traffic to/from another specific network, you and the other network can both save Internet Transit costs by exchanging traffic locally, i.e. peering.  Of course an enormous amount of your traffic is directed to Google.  If you have a presence in any data center where Google has a presence, you would love to peer with Google, as that saves an enormous amount on your payments for upstream Internet Transit.

A similar effect plays out among Tier 1 providers.  If one tier 1 network cuts a special deal with Google, Google routes all their traffic through this provider and suddenly the other tier 1 networks have large asymmetries in their tier 1 peering arrangements.  Either they also cut deals with Google or they have to renegotiate their tier 1 peering arrangements to pay for the traffic asymmetry (something that's highly unlikely!).  Google is the one with leverage here!

I don't know what, if anything, Google pays for bandwidth, but it's not paying $360M for Internet transit.  Sorry Credit Suisse, you better go back to analyzing derivatives, credit swaps and other purely financial plays.

Google does have costs.  They have data centers in many parts of the world and they have a private fiber backbone that interconnects their sites and connects their private network to many, many potential peering points.  Operating their private backbone is a real cost to them and I haven't examined their financial reports to see if there is any way (from public data) of estimating their costs for this private network.

But until someone does this analysis, forget what you've read from Credit Suisse.

Best conference bet for 2009 ― eComm 2009, March 3-5, San Francisco

I attended a number of conferences in 2008, both interesting and not so interesting.  One conference stands out, for the range of interesting speakers and the variety of interesting people I met.  That was the first Emerging Communications Conference, eComm 2008, organized by Lee Dryburgh.  Many of talks from this conference are available on Slideshare and as podcasts on IT Conversations.

eComm 2009 is scheduled to take place at the San Fransico Airport Marriott, March 3-5, 2009.  I highly recommend you check it out.

This is not a trade show with vendors hawking today's products and multiple tracks full of vendor product pitches. 

Presenters have been chosen for the quality of their proposals:  is it new?  is it disruptive?  what will the audience learn?  (As an adviser, I've been in on those discussions).  Like last year, the format is one track spread over three days, with 15 minute presentations, 5 minute lightning presentations, panel discussions and social time.  It all adds up to a veritable fire hose of information.

There's a list of speakers here.  Major topics for 2009 (so far) include:

* Mobile Social Networking (MoSoSo)
* Open Handsets & the Open Ecosystem
* Both Voice and Video Evolution
* Convergence of Media with Personal Communications
* Open Spectrum
* Open Communication Platforms
* Leveraging Cloud Computing
* Social Computing
* Towards 4G Wireless
* P2P and Decentralization of Telecoms
* Communications enabling business processes, especially B2C
* New Forms of Contactability and Connectability
* Emerging Markets

And last, but by no means least, if you mention my name you get 20% off.  More specifically, if you enter the promo code "BroughTurner" (case-sensitive) at the appropriate point during registration, you'll get 20% off the registration fee.  This works now, while early bird rates are in effect, and I'm told it will also work right up to the last minute ("late", not on-site registration), although then it's 20% off the full conference rate, and only if the event is not sold out!

I hope to see you there.

A reason why Asia leads in mobile apps and is advancing with Internet apps

A picture is worth a thousand words.  Thanks to Ben Joffe for this picture, which is slide 21 of his presentation, The Digital Silk Road.

Of course, this is a simplification, as it misses China. 

China doesn't have 3G yet but they have more mobile phones than any other country and they are fast to adopt what can work on 2.5G.  For example, the Chinese adopted ringback tones very very rapidly after the application emerged in Korea -- years ahead of Europe or the US.  And, even at 2.5G speeds today, China supports a rich variety of mobile applications.

Likewise, while China's relative proportion of Internet connections above 5 Mbps is relatively small, there are more broadband Internet connections in China than in the US and China is adding new fiber Internet connections faster than any other country in the world.

The Mobile Web: Limited But Getting Better

I had an interesting discussion with Katrin Verclas of MobileActive.org as an event last month covering a range of topics including the prospects for mobile web access in emerging markets.  As a result, she and Patty Mechael pressed me to write up my thoughts on the mobile web today and over the next few years.

Very roughly, I'm depressed about today's still prevalent walled gardens, but very optimistic that open mobile access to the Internet will become available in coming years.  You can read my reasoning here.

Packet inspection renders Internet connection unusable for blogging

Here's one I hadn't run into before.

The typical hotel broadband access service checks the MAC address of your computer. When you first connect, it sees the new MAC address, interrupts whatever access you were about to make and instead delivers a sign-up screen.  After you've signed up, your packets are routed directly to and from the Internet until your sign up period expires.

Of course this means every packet is being inspected to verify the MAC address is for a computer that has paid for access.  This is very light packet inspection, nothing like the DPI that various ISPs have been called to task for.  It's also very fast so it doesn't introduce noticeable latency, at least until now.

Last night I was staying in the InterContinental Nehru Place in Delhi and I observed a strange effect on perhaps 1 out of 3 or 1 out of 4 page loads.  First I would see a very brief splash screen from the hotel's ISP, then my desired web page would load.  This was unusual but not a bother.  Next, while using Bloglines, every now and then the left screen list of unread posts would fail to refresh, instead displaying a message about access had timed out.  However the next refresh always worked, so again, I can live with that.

Finally I tried to create a blog post but I lost the text as I went to post it.  Ugh!  Apparently, Typepad's blog author's content posting page requires reasonably fast turn around time for update messages.

Clearly MAC address redirection was beginning and then being canceled.  I'd never run into anything like that before -- a case where even shallow packet inspection is a problem.

Visualizing African Internet Connectivity

Here is a great illustration of the not-so-fast or reliable connectivity at African Universities as seen from Trieste, Italy.  Only 46 seconds.

Thanks to Marco Zennaro at the Science Dissemination Unit (via TIER at UC Berkeley).

Similar motivations for mobile and Internet usage in rural Zambia

Over the years I've uncovered a number of serious studies of the social and economic impact of mobile phone adoption in emerging markets.  In fact, my list of links (embedded in various blog posts: 1, 2, 3, 4, 5, 6, 7, 8) has become a reference for several people, for example here. I've been somewhat less organized regarding the social and economic impact of Internet access, but I've resolved to track such references as well. 

Here's something I stumbled on this weekend -- a study by Paula van Hoorik and Fred Mweetwa of TNO Information and Communication Technology in the Netherlands, entitled "Use of internet in rural areas of Zambia."

What struck me in this study was the parallels between the social and economic benefits they found and those conventionally associated with the adoption of mobile phones.

Based on our study, the most important social benefits are:
• Internet enables people to keep their network and enlarge it by communicating with friends, family and others
• Internet enlarges the world of people in rural areas by giving access to information
• Internet brings knowledge and supports education

The most important economic benefits are:
• Reduction of commute time
• Saving money in a lot of different ways, such as only traveling to pick up something when you know it is actually there instead of having to return multiple times
• Bringing new opportunities and using them, such as learning new farming methods or opening an internet café to make a living.

Most studies of mobile phone adoption in emerging markets, especially amongst the poor, show that social reasons - communicating with friends and family - come first, economic advantages come second.  And among both social and economic advantages, the ability to eliminate journeys, i.e. save time by not having to walk to the next village to ask a question, is extremely important.

I suppose I shouldn't be surprised.  At root, they are both communications technologies and our use of such technologies is driven by basic human needs that are remarkably similar, for everyone, everywhere. 

How tiered Internet pricing could actually facilitate P2P

Time Warner Cable's planned experiment with tiered charging for Internet access has generated a flurry of coverage in the blogsphere, but no new insights (at least that I've seen).

The primary problem ISP's complain about is that 5% of their customers use 90% of the available bandwidth and when they examine this traffic, it's mostly peer-to-peer file sharing.  A reasonable question is how to allow as much of this traffic as possible without increasing an ISP's variable costs or slowing down their other users.

This may not be as difficult as it appears.   Indeed if Internet access was as competitive as mobile telephony, we might already have seen what I'm about to propose — a combination of bundled pricing equivalent to mobile's "free nights and weekends" and "free on-net calls" with a way to facilitate P2P traffic that leverages exactly these "free" periods.

An ISP's costs

ISPs have some costs which are relatively fixed and others that are tied to usage.  A network is a relatively fixed cost and when it's not full, the incremental cost of adding traffic is zero!  This is the reason mobile operators give away free nights and weekend.  They've built their mobile network for the peak daytime traffic, so it costs them nothing to run promotions that add incremental traffic at off hours.  Peak hours and off hours may be different for an ISP, but the concept is the same. When a data pipe is lightly loaded the ISP's cost of adding incremental traffic is zero.

On the other hand, some ISP costs are usage based, for example "IP Transit" or more properly, Internet Transit.  This is the ISP's upstream cost to send and receive traffic to/from the rest of the Internet.  However, even here, usage-based costs occur at heavy usage.  Light usage periods don't save money.  To understand what's happening, it's worth a digression on Internet Transit.

Internet Transit

Internet access is monopoly or duopoly or a heavily regulated industry.  The middle mile connections from the local network to the Internet backbone may or may not be competitive depending on where you are.  But the Internet backbone itself is extremely competitive.  If you can get to a major Internet Exchange Point in the US or Europe, there are many providers offering extremely competitive rates for Internet Transit.  Typically these services are priced on a megabit per second per month basis (Mbit/s/Month) with lower rates for higher volume commitments.  The other key idea is that charges are based on the 95th percentile of all the five minute data rate samples taken during the month.  So an ISP can have a few bursts above their typical rate, as long as they represent less than 5% of the sampled intervals.

But this also means there is no extra cost to run at or near the typical rate at all times.

Local traffic

Even more important, if file sharing is done with other computers on the same ISP's network, then there is no need to pay for Internet Transit at all.  The question is how to figure out which potential peers are "on-net" and which are "off-net."

Sending signals to P2P software

Most P2P file sharing software has relatively little knowledge of locality.  Some P2P software practices "prefix awareness," for example, Joost gives preference to peers in the same /24 IP address block when they are available.  But if a major operator provided an automatic way for P2P client software to determine whether a prospective peer's IP address was currently reachable "for free", it seems likely the file sharing community would leap on it, and if there's money to be saved, active file sharers would download the new clients immediately.

A standard way to present such information might be via an extension to the XML-based response codes in one of the whois information exchange proposals, e.g. from ICANN or from APNIC.  Also, while what I'm proposing might start as a pricing plan rather like a mobile operator's "free nights and weekends" and "free on-net calling," it's not hard to see extensions where an ISP could offer dynamic access to underused capacity to those programs that were prepared regularly interrogate an ISP's server and use just the advertised off-hours capacity.

In closing

People liked fixed price deals.  Unlimited is great, but there's plenty of experience with bundles of minutes and the idea of data bundles has already showed up in 3G mobile data plans.  The combination of several tiered data bundle prices with the availability of "free" connectivity for "on-net" peers and during off peak intervals is likely to appeal to file sharers and produce better results for both the sponsoring ISPs and file sharers alike.

The Moral Instinct, cyberjustice and Internet law

There's an excellent article in today's NY Times Magazine, The Moral Instinct, in which Steven Pinker summarizes recent neuroscience research on the human moral sense.  This bears directly on how we think about justice and morality in cyberspace.

He covers the enormous progress (at least since I was in college) in our understanding of where the human moral sense comes from, including what is universal, what is cultural and how we can be easily mislead by "an aura of sanctity, distracting us from a more objective reckoning of the actions that make people suffer or flourish."

Illusions are a favorite tool of perception scientists for exposing the workings of the five senses, and of philosophers for shaking people out of the naïve belief that our minds give us a transparent window onto the world (since if our eyes can be fooled by an illusion, why should we trust them at other times?). Today, a new field is using illusions to unmask a sixth sense, the moral sense. Moral intuitions are being drawn out of people in the lab, on Web sites and in brain scanners, and are being explained with tools from game theory, neuroscience and evolutionary biology.

While the article doesn't mention the Internet, a better understanding of our moral sense can only help temper our approach to law and culture in the new social relationships facilitated by the Internet.  Depending upon the cultural setting, subjects as diverse as ethnic humor, pornography and religious discussion can be completely acceptable or morally outrageous and yet, except for some language effect, the Internet cuts across cultural boundaries.  Then there are Internet-related tragedies like the death of Megan Meier which provoke discussion and calls for sweeping laws to regulate cyber-behavior.

Undoubtedly, our laws and culture will evolve in response to the Internet age, but hopefully we do this with care, taking advantage of what we can learn from history and from the new science of the moral sense.

Here are a few short quotes from Pinker that struck me:

There are many other issues for which we are too quick to hit the moralization button and look for villains rather than bug fixes.

Our habit of moralizing problems, merging them with intuitions of purity and contamination, and resting content when we feel the right feelings, can get in the way of doing the right thing.

Far from debunking morality, then, the science of the moral sense can advance it, by allowing us to see through the illusions that evolution and culture have saddled us with and to focus on goals we can share and defend. As Anton Chekhov wrote, “Man will become better when you show him what he is like.”

If you are at all interested in morality, neuroscience or evolutionary biology, I highly recommend the Pinker article.