Updated post: Level 3's Lower Cost Comes From Owning The Network, Not Free Peering
Yesterday, Level 3 went public with a statement saying that Comcast was for the first time demanding, "a recurring fee from Level 3 to transmit Internet online movies and other content to Comcast’s customers who request such content." Level 3 is saying such actions by Comcast are at the heart of the network neutrality debate and as one would expect, we've seen a great deal of thoughts, posts and comments about this whole subject in the past 24 hours.
Some posts have done a really good job of educating readers on how things like settlement free peering work and have brought to light how content is delivered on the Internet and what some of the relationships amongst carriers and MSOs look like. Other posts I have read have strayed far off the subject, in some cases accusing Level 3 of "stealing" bandwidth from Comcast and many other posts simply want to make this whole issue out to be about money, or Netflix. There are a few points that I think are important that I haven't seen addressed and are what I feel the real discussion should be about.
For starters, this is not simply about money. While Level 3 said that have, "agreed to [Comcast's] terms, under protest, in order to ensure customers did not experience any disruptions", the actual terms of the deal and the money being exchanged between the companies is not substantial. Everyone assumes we're talking big dollars here, but we're not. The real argument by Level 3, which in my mind is fair, the idea that last mile providers are asking for a payment when there is absolutely no competition and no other options for Level 3 if they say no. There is no one else that Level 3 could buy something off to get access to Comcast's eyeballs. Comcast could make any rules they want, and if Level 3 wants to continue to distribute their customers content to Comcast eyeballs, Level 3, or any other carrier, would have to agree to Comcast's demands.
This isn't two commercial companies having a commercial discussion because the discussion is completely lopsided. And while Comcast is not gouging Level 3 today and the two companies aren't really fighting as many make it out to be, Level 3's point is that this could become a problem down the road and they feel someone has to stand up to it. Level 3's argument is that the FCC should establish guidelines on how these relationships should work. Also, Level 3 is not "pushing" traffic to Comcast. This is content being requested by Comcast's own subscribers, and being pulled from Level 3. The idea that Level 3 is simply trying to dump all of this traffic onto Comcast's network is laughable. It's Comcast's traffic.
I've seen some suggest that Level 3 should just say no to Comcast, but that's not a realistic approach since that immediately puts all of Level 3 customers, who are the content owners, in jeopardy of not being able to get their content to the consumer, who is Comcast's customer. Of course some are making this whole story out to be about Netflix, but that's not what this is about. If Level 3 had not announced their new contract with Netflix to distribute their content, no one would even be mentioning Netflix in this story. This about the underlying principle of just how much control any last mile provider should be allowed to have and whether or not they should be allowed to prioritize traffic.
Of course when you bring the whole NBC subject into the picture, then this gets even more interesting with some suggesting that if the deal with NBC goes through, Comcast could give the delivery of their content more priority over another content owner, from another carrier. On Comcast's blog, the company was quick to say that, "Level 3 has inaccurately portrayed the commercial negotiations between it and Comcast. These discussions have nothing to do with Level 3's desire to distribute different types of network traffic." That may well be the case - today. But the real explosion of traffic on the Internet is from video, so while Comcast is not specifically calling out video related content from Netflix or anyone else, that's really what we we're talking about.
Another really big issue that seems to be missed in this whole discussion is that for the traffic that's moving, there is no change to Comcast's cost base. Level 3 may be sending Comcast more traffic, due to Comcast's customers demanding it, but that does not mean it costs Comcast more money to deliver it. I simply see Comcast as using this as an opportunity to try and charge Level 3 money, hoping that folks won't really ask what the real cost impact is on Comcast. You will notice that in the Comcast post on their blog where they commented on Level 3's statement, nowhere did Comcast say that the additional traffic from Level 3 was costing them more money. And if it does, it's Comcast's burden to bear as it's content that their customers are demanding. Comcast is basically asking Level 3 to subsidize their service by charging Level 3 a fee and this is where things become scary if Comcast is allowed to get away with this.
What I don't see Comcast talking about, or anyone else suggesting, is the multiple ways that Comcast could work with Level 3 to help alleviate the traffic and cost on Comcast's network. One way to do this would be for Level 3 to deploy deeper into the Comcast network, which would help alleviate the issue, yet I'm hearing that Comcast isn't looking at this as an option. If Comcast expanded its peering and improved its internal network and worked with Level 3 to allow them to be deployed deeper in their Network, Comcast could even deliver better service while lowering its costs. The idea that they only way Comcast can combat this is to charge Level 3 is simply not the case. If there is an imbalance in traffic, like Comcast suggests, then why isn't Comcast allowing Level 3 to carry the traffic further into their network to equalize the cost? No one at Comcast seems to be willing to answer that question.
Due to Comcast's actions, Level 3 says it is, "approaching regulators and policy makers and asking them to take quick action to ensure that a fair, open and innovative Internet does not become a closed network controlled by a few institutions with dominant market power that have the means, motive and opportunity to economically discriminate between favored and disfavored content." From my perspective, I'm glad to see Level 3 making this issue public and bringing it to everyone's attention, as the topic needs to be debated. The discussion should not be about Netflix or the CDN business as that's not what the crux of this is about. Net neutrality is really the heart of this debate.
(Disclaimer: I'm a MSFT employee and work in this field, etc, etc, but my remarks are my own, not my employers'.)
Dan, I was with you until your very last sentence. Yes, this is about highlighting the last-mile issue, and yes, this is about LVLT taking a stand, IMHO. (No personal knowledge of the situation.) But like so many others, your last point conflates "net neutrality" with "the last-mile monopoly".
You said it exactly yourself: "There is no one else that Level 3 could buy something off to get access to Comcast's eyeballs." THIS is the problem -- last-mile customers have no competitive options! Solve that problem, and this "net neutrality" debate vanishes.
Posted by: Jason Sherron | Tuesday, November 30, 2010 at 02:32 PM
Hey Jason, agreed. The last mile problem creates the net neutrality issue. You solve that problem, it does vanish. Do you think I need to highlight that point more? Maybe I didnt make it very clear. Thanks.
Posted by: Dan Rayburn | Tuesday, November 30, 2010 at 02:36 PM
" The real argument by Level 3, which in my mind is fair, the idea that last mile providers are asking for a payment when there is absolutely no competition and no other options for Level 3 if they say no. There is no one else that Level 3 could buy something off to get access to Comcast's eyeballs."
They could buy transit from anybody who peers with comcast, or anybody that has paid peering from comcast. In short, they could purchase internet access from anybody that carries a full routing table.
Posted by: Matt Levine | Tuesday, November 30, 2010 at 03:06 PM
Hi Matt, the way Comcast has engineered their network, Level 3 can't simply buy transit from anyone who peers with them.
Posted by: Dan Rayburn | Tuesday, November 30, 2010 at 04:05 PM
"Hi Matt, the way Comcast has engineered their network, Level 3 can't simply buy transit from anyone who peers with them."
Are you speaking in terms of policy or technology? In terms of policy, the comcast peering policy simply states that you can't "just" sell comcast routes to customers..
In terms of technology, are you implying that comcast would somehow prefix filter level3 source IP's from say, their global crossing interconnects? That would break anyone multihomed out of Level3's IP space.
Posted by: Matt Levine | Tuesday, November 30, 2010 at 04:21 PM
Good Article. I would like to hear more details if you were willing, about how LVLT fits into the Comcast infrastructure. For example, how close to the end user does LVLT get today when it dumps the Netflix traffic on to the Comcast system? How (with the permission of Comcast) could it go deeper? I think a lot of people would appreciate your insight on this.
I disagree though that Comcast is not worried about the cost. Level 3 in a recent presentation claims one hour of video requires 20,000 times the number of bits that web browsing does. Comcast internet performance is already far from optimal. They will have to spend to stay ahead of the performance curve.
I think it is mostly about the failing Cable TV model. Comcast knows that people are more apt to spend $8 for Netflix than $12 (or whatever it is for HBO). People want to have control of what they can watch.
Posted by: Peter | Tuesday, November 30, 2010 at 05:12 PM
Dan, you said "The real argument by Level 3, which in my mind is fair, the idea that last mile providers are asking for a payment when there is absolutely no competition and no other options for Level 3 if they say no."
Frankly, I'm puzzled why people believe that this is a policy issue, and more specifically the FCC's domain. This scenario seems like a "restraint of trade" dispute that the U.S. Department of Justice would routinely explore.
Why are new laws or new policies required?
Posted by: David H. Deans | Tuesday, November 30, 2010 at 06:34 PM
Dan, you are absolutely dead on. Too many people think this is about Netflix or say this is what Level 3 does to folks like Cogent, by charging them, yet Cogent is NOT an last mile provider. It's not the same thing and Level 3 has a really valid point.
Posted by: Steve M. | Tuesday, November 30, 2010 at 11:37 PM
>If Comcast expanded its peering and improved its internal network and worked with >Level 3 to allow them to be deployed deeper in their Network, Comcast could even
>deliver better service while lowering its costs.
Comcast is notorious for not peering with anyone. Comcast wants to sell access to their
customer base, not give it away. I know of several networks with tens of gigabits of
traffic to/from Comcast who can't get settlement free peering.
Posted by: Ed Gein | Wednesday, December 01, 2010 at 02:44 AM
This dispute is identical to those over the charges content providers charge satellite and cable providers today such as Fox and Cablevision or Comcast and DISH. It is a commercial dispute. Settlement charges have been there since the beginning. Take them away and the Internet goes away. If Netflix underpriced its service because it assumed free delivery of its content, that is their problem. I realize that Comcast could raise its rates to everyone so that Netflix customers can watch more movies, but this is then the same as the bundling issue on cable: making people pay for something they do not use.
Posted by: Bill | Wednesday, December 01, 2010 at 06:51 AM
Here is a dispute between two companies about the content. Level 3 made a statement they want justice, the issue is not related with money. Network neutrality is a principle proposed for user access networks participating in the Internet that advocates no restrictions by Internet service providers and governments on content, sites, platforms, the kinds of equipment that may be attached, and the modes of communication. Net neutrality is really the heart of this debate.
Posted by: Account Deleted | Wednesday, December 01, 2010 at 06:55 AM
Dan is correct; this is about Net Neutrality and power. Interesting to note the political background:
- Level 3’s strong regulatory savvy (backing of current admin.).
- Level 3’s legal head is retiring this month, final outgoing act.
- FCC Chairman presenting today on NN, needed a live example.
Posted by: Jim | Wednesday, December 01, 2010 at 09:06 AM
Dan,
This really is an issue about money, the money part is just further downstream.
And in order to get to the money part, you have to go through the power part first.
I doubt that if there were not significant $$$$$$$ downstream that Comcast would behave this way.
Posted by: Kevin | Wednesday, December 01, 2010 at 09:06 AM
Dan, I blogged about this in the Journal yesterday. You have identified most of the issues here. However, there is a deeper, more sinister strategy at work by Comcast and that is to slowdown or impede the delivery of video over the web. This is because it potentially cuts into their legacy cable tv business. These are cable guys at the end of the day and they only know cable. Comcast has ventured beyond the cable to the home and tried to make content acquistions so they deserve some credit. At the end of the day, they are cable guys and they will do whatever it takes to protect that franchise. The whole "bandwidth" argument is the industry crutch that they pull out every time someone encroaches on their fiefdom. The cable companies have purposefully delayed the development of video over the web. They could have led this by leaps and bounds if they wanted to. Because they couldn't control it, like they control the lines to cable TV, they had no incentive to see this move forward. Technologically, they have the capability to improve and facilitate the flow of content/entertainment to the home but they continue to put up obstacles, both visible and invisible. They have a virtual monopoly (despite satellite and telcos) and they want to protect it at all costs. One final point, they don't want any competition in what they do. Just look at the way cable companies have gotten together and have carved out the U.S. television market.
Posted by: Carlos Haugabook | Wednesday, December 01, 2010 at 10:18 AM
hello Dan,
In my opinion, it's more a question of revenue sharing
between content owners like Netflix,Google, etc.. and Telcos like comcast and national telcos in main countries.
it seems logicalthat the telcos must be getting jaleous when they see all dollars flying into the content owners pockets instead of theirs.
This problem will in my opinion appear more and more everywhere in the near future.
I believe that solutions like cedexis.com who allow the uses of multiple CDNs based on performances have a bright future.
Damien,
Posted by: Damien Wetzel | Wednesday, December 01, 2010 at 10:20 AM
These public spats over peering come and go- has anything ever actually changed as a result?
Posted by: Steve Lerner | Wednesday, December 01, 2010 at 10:28 AM
"Dan, you are absolutely dead on. Too many people think this is about Netflix or say this is what Level 3 does to folks like Cogent, by charging them, yet Cogent is NOT an last mile provider. It's not the same thing and Level 3 has a really valid point."
Steve: Um, there's quite a lot of high-speed office connectivity single-homed to 174.
Posted by: Matt Levine | Wednesday, December 01, 2010 at 11:21 AM
What if any single ISP worlwide providing internet access ask/demand to the carriers, the CDN's and the content owners pay a fee for IP transit... besides of end users montly payment for broadband access services..
..so the two-side-revenue model in place!
I think that will lead a very large ISP - Carrier - Content Owner consolidation...
No doubt ISP - carriers - Content owners have to move on a set of rules who goverments and regional Internet authorities have to define and enforce...
btw... it would be possible?
Title: Who Controls the Internet?: Illusions of a Borderless World
Author: Jack Goldsmith, Tim Wu
# Publisher: Oxford University Press, USA (June 30, 2008)
# ISBN-10: 0195340647
Title: The Future of the Internet--And How to Stop It
Author: Jonathan Zittrain
# Publisher: Yale University Press (March 17, 2009)
# ISBN-10: 0300151241
Title: Code: And Other Laws of Cyberspace, Version 2.0
Author: Lawrence Lessig
Posted by: Martin Ortiz | Wednesday, December 01, 2010 at 03:17 PM
The next logical extension of this is for Comcast, et al, to send any content provider, including an arbitrary website, a bill for the data Comcast customers requested from the site. If you, as a content provider, don't pay then your traffic is blocked or given a lower priority.
Posted by: Tim Walker | Wednesday, December 01, 2010 at 09:18 PM
SOME BIG ERRORS IN THIS
"Hi Matt, the way Comcast has engineered their network, Level 3 can't simply buy transit from anyone who peers with them."
Level 3's competitors have the EXACT same traffic and have multiple ISP connections to deliver to the Comcast network. The only thing Level 3 can't do is deliver it for FREE through any of the Comcast peers. The Internet is a fair playing field when you play by the same rules you hold others too.
"Hey Jason, agreed. The last mile problem creates the net neutrality issue. You solve that problem, it does vanish. Do you think I need to highlight that point more? Maybe I didnt make it very clear. Thanks."
Also highly inaccurate. The last mile issue is the same with Level 3 as well. Cogent MUST purchase from Level 3 to reach Level 3's customers.
The only thing new here is Level 3's massive growth in CDN business dominating their entire traffic profile and changing their company for the future. Level 3 wants to be a settlement free Tier 1 while shifting everything to the CDN business.... Sorry Level 3, you need to pay for your bits if you are not a balanced network.
Posted by: Anon Poster | Saturday, December 04, 2010 at 05:33 PM
December 7, 2010
Dear Comcast,
On July 27, 2010 Starcraft 2 "A GAME" came out with a midnight release. Comcast in my area and surrounding area was down for maintenance. I then wrote an E-mail to you which was never responded to. In this E-mail I wrote about taking major internet events into consideration when turning connections off. Tonight conveniently my connection is once again down in the area as stated by a pre-recorded message. Yesterday comcast was also down for several hours. Oddly enough World of Warcraft Cataclysm came out tonight. Now although I do not play World of Warcraft I find it interesting that your network goes down when the midnight release occurs and when pre-order download is open. Can't your network handle the amount of customers you have? Can't you put off maintenance off for a day? What you are doing to gamers is similar to turning off the Superbowl, world series or what ever else sports lovers like to watch. If this happens again when a major game title is about to launch I will be switching to AT&T without hesitation. I like Comcast's speed but the instability is becoming unbearable. I would much rather have a stable slower speed than a fast connection that goes down all the time.
Sincerely,
Matthew A. thorns
Game Designer
P.S. Below is the original E-mail I sent in July. I am currently on my neighbors AT&T wifi connection and I am still down.
Dear Comcast,
I am writing this on a terrible dialup connection to make a suggestion to you for your future maintenance and update. Please forward this to anyone that is willing to listen. I am a victim of an outage that occurred around midnight on July 27th. I strongly suggest that in the future you cease all updates and non repair maintenance during significant online events. Tonight StarCraft 2 from Blizzard came out and I am unable to play the next several days due to working a full time job and also going to school full time. I am very displeased and I thought a company that supplies internet to millions of subscribers would acknowledge the gaming community. Here are a few suggested games that will most likely be a big deal to the gaming community in the near future. Diablo 3, Portal 2, world of warcraft cataclysm.
Sincerely,
Matthew A. Thorns
Game designer
Posted by: Comcast Doesn't Care About Gamers | Tuesday, December 07, 2010 at 02:04 AM
Vvery informative! thank you very much.
Posted by: john andrews | Thursday, December 09, 2010 at 09:31 AM
The last mile imbalance works equally both ways. While Comcast is the only way Level3's customers can reach Comcast's eyeballs, Level3 is also the only way Comcast's eyeballs can reach Level3's customers. If Comcast's customers can't reach the content providers they want to reach, Comcast is going to have a hard time keeping its customers happy.
Posted by: David Schwartz | Wednesday, December 15, 2010 at 01:23 AM
I am not a big defender of Comcast because I agree that its business model deserves to get torched if Netflix gains share of video entertainment spending in coming years. There is no question in my mind that Netflix launched this service at this price because Level Three is only charging $0.06 - $0.09 per HD stream. Level Three's pricing was based on no cost to access Comcast customers and no strategic concern about balanced traffic at peering points. IMO, this debate is simply a case of cross subsidization. Old school Comcast High Speed internet and cable subscribers are cross subsidizing early adopters of Netflix streaming. If everyone switched to Netflix, the last mile network congestion will degrade service or require significant capital spending necessary to meet the peak load requirements of maintaining multiple simultaneous streams. Am I missing something?
Posted by: Dave Brown | Tuesday, December 21, 2010 at 09:11 PM
Absolutely agree, Comcast is just being stubborn with their policies, I don't know if they make more money with it, but if they actually developed this they could definitely offer a better service.
Posted by: George Lewis | Sunday, April 17, 2011 at 04:35 PM