Basic truths about broadband speeds (reaction to Kevin Walsh on GigaOm)

On GigaOm Kevin Walsh has written down some basic truths about broadband, as he sees them. He sees a big problem with politicians wanting the wrong things when it comes to guarantees on bandwidth speeds. As he says:

Many believe that broadband service providers selling, say, a 5Mbps service should be required to set aside the same amount of capacity in order to fulfill that implicit service-level agreement (SLA). In other words, if you pay for 5Mbps, it’s there when you need it. But the reality is that networks, just like hotels and airplanes, are almost always oversubscribed — the owners of these assets sell more capacity than they have available.

Unfortunately this article messes things up just as badly as the politicians criticized. I reacted in the comments and post the comment here too. There are the important elements that need considering.

1. The speed of the line from the end-user to the local aggregation point (switch, cable head-end). DSL and wireless technologies are particularly crappy, as the speed the end-user can have is a function of the distance the end-user is from the DSLAM or the antenna. Unfortunately there are still telco’s that sell up to 8mbit or up to 20mbit subscriptions that can only attain 4-8mbit/s because of distance limitations. The really nasty one’s try to upsell the customer to a top tier 20mbit/s line where only 4mbit is achievable and so a 4mbit/s subscription would have sufficed
2. The speeds that can attained between 6pm and 10pm and the speeds that can be attained between 2am and 6am. These can be limited by oversubscription on:

• a. the local segment (cable and wireless) and
• b. on the ISP’s WAN
• c. from the ISP to the rest of the world

Oversubscription on the local segment is a fact of life on cable and wireless networks. It is a shared medium. This should be clear to end-users. It is not bad, it is a fact of life. It’s effect is that between 6pm and 10pm the speeds can be erratic. between 2am and 6am the listed speeds can quite often be attained.

Oversubscription on the WAN is part of the problem you’re describing above. Oversubscription on the WAN is not an economic fact, it is a result of crappy network planning. Oversubscription on the WAN can be completely unnoticed by the end-user if the network operator builds enough bandwidth into it’s WAN. If the network owner and the ISP are the same entity, this shouldn’t be a problem. With traffic growing 50% per year proper network management dictates that an oversupply of bandwidth is necessary anyways. Statistics from the AMS-IX in Amsterdam show that peak traffic is about 50% higher than average and three times higher than the bottom. So next year your average is the same as today’s peak and in 2.5 years even the bottom is at today’s peak. WAN Bandwidth is a problem in some countries like the UK and the USA where the costs of backhaul to and from smaller communities are extremely high because of regulatory and/or competitive problems. Mind you, technical and cost limitations are often not important here, as the costs of installing faster equipment is often not prohibitively high. (DWDM, 10Gbit/s ethernet etc) This also means that you don’t have to build your network in a fashion where everyone can achieve max speeds at the exact same moment. Just carefully planning it. Like the highway system, where we don’t expect all car drivers in the US to show up at the Brooklyn Bridge at the same moment (or to start driving at the same moment at all, regardless of location).

The costs of traffic from the ISP to the rest of the world is governed by the economic laws of peering and transit. For an explanation see my article on Ars Technica. Whether enough is available to the end-user is dependent upon how easy it is for an ISP to get peerings with the most important networks (Google, Microsoft, Yahoo, Akamai etc) and the local costs of transit. Many developing nations find that their biggest problem lie here:

• The national incumbent monopolizes transit traffic and charges outrages amounts for it.
• No local internet exhanges to keep local traffic local.
• No possibilities for local peerings with Google, Microsoft, Yahoo etc meaning that the transit link gets hit harder.) 

Amsterdam, London, New York are places with low costs for transit ($4/mbit/s/month) and many peering opportunities, so any network operating there should be able to get enough traffic for their end-users. Dave Farber once mentioned that traffic costs were only between 1%-5% of a subscription.
So, to conclude:

• Politicians are right to complain when listed speeds on the local loop cannot be attained because of distance problems. Providers of DSL and wireless should be put in the doghouse for this. The should inform their customers properly of what speeds can really be achieved.
• Cable networks and wireless networks could be required to publish the mean and median speeds users can attain between 6pm and 10pm.
• Problems on the WAN and on the interconnect to the rest of the world are either a result of bad investing in backhaul or because of regulatory and competition problems. If that is the case the ISP should inform its customers of the situation, explain why this is the case and show how it deals with distributing a scarce resource among all the users. A good example is the Plusnet DSL network in the UK who are very clear on how they prioritize network traffic between different classes of customers.

Mobile termination in non CPP countries

As I only discovered two days ago, Ofcom is busy with a consultation on the future of mobile termination charges.http://www.ofcom.org.uk/consult/condocs/mobilecallterm/

On the site you can find a report by Analysys Mason that describes the situation with regards to mobile termination tariffs in Canada, HongKong, Singapore and the USA. http://www.ofcom.org.uk/consult/condocs/mobilecallterm/annex8_1.pdf
One of the interesting conclusions is, that for elements like penetration it doesn't really seem to matter which tariff scheme is used, but it does affect the usage of mobile telephony.

Also have a look at Sri Lanka, the only country to have decided to go to CPP, but decided not to after a wave of protest in the country on the issue. http://www.trc.gov.lk/images/pdf/cppeng.pdf their approved retail tariffs are interesting to look at too. http://www.trc.gov.lk/press-room/approved-tariffs/91-sri-lanka-telecom-plc.html

(Mobile) termination rates may end your career

I have been quite critical on "termination charges". One reaction I received to my previous post via e-mail was quite eye-opening.

I have a friend, he was a member of the board of the regulator in XXXX. He ‘was’ because next day after regulator presented decision to lower MTRs, parliament (instead approving this decision) voted to change the CEO and the board. He was a member of dispute resolution commission. While resolving an ordinary dispute one of the sides started to influence him. Finally some people were trying to shut him. That’s more or less realistic picture of regulatory environment in developing country.

As a Dutchman I don't know what corruption is. It doesn't really exist here as far as I know. Maybe I'm blind, but we're doing pretty well according to Transparency International and World Audit Corruption. However, most countries are fighting a 2 TRILLION dollar industry (services (fixed and mobile)and equipment), when they want to change the rules to make sure that their country benefits from (mobile) telecommunications. To see some recent examples look for instance to these postings by Steve Song about telecoms in Africa, or at Russia aiming to ban foreign VoIP. Benefiting often means more for the same money, or more for less money and this is perceived as a loss by the local telecommunications companies.

You know, it's an increase in telecommunications usage that benefits economies, not an increase in telecommunications revenues, but that is awfully easy to say from the Northern bit of Europe

Low (mobile) termination costs are good for the poor

I've been interested in mobile termination rates for quite a while and I've blogged about it previously. In recent days I've been looking at it again for a variety of reasons and I found some new data, which might are interesting in the debate and that show that low termination rates or no termination rates are beneficial to poor people and to competition in general. I also read this article by Emma Buckland or Analysys Mason, who claims that in developing countries, where almost all mobile users are prepaid, a move to MPP would be even less popular.
But first, what was it all about again.

In most countries in the world, if you call a telephone number, your network will pay some money to the receiving network for the termination of the call. This system is called, Calling Party Pays. The effect is that in those countries the receiving of calls is free. The amount that needs to be paid (termination rate) is often determined by the regulator. If the market would set it, it would result in termination monopolies and as Orange in the The Netherlands once showed, you can raise termination costs without affecting incoming call levels, so it's essentially free money. The height of the termination rate is very contentious. Even more so because many regulators have deemed it right that some networks should receive more than others (asymmetric). Incumbents less than new entrants and mobile more than VoIP and fixed lines. Termination rates can be anywhere between $0.004 cents per minute (India) to $0.20 cents per minute (calling a mobile in Bulgaria). For a nice overview of the EU, see these graphs of the ERG.

There are a few countries that have a system that is known either as Bill and Keep or Receiving Party Pays or Mobile Party Pays. The way this is generally implemented is that the networks of the calling parties do not pay each other at all for the traffic terminated on the other's network. (I don't know of any country that has implemented a system that works like a collect call, where the receiving parties network pays back to the originating parties network) Countries that have this model are the USA, Canada, Sri Lanka, Singapore. There isn't one effect of this model, as in the USA you buy a bucket of minutes and whether it's an incoming or an outgoing call, your bucket is emptied) but in Singapore and Sri Lanka incoming calls are free.

There are many countries who have moved away from this model to CPP (like France and China). Sri Lanka had decided to move to CPP, but reversed that decision after an uproar in the country. So you would expect CPP be better for competition, better for consumers etc. And this is argued by the GSMA. And well, that just isn't true. The exact opposite is the case. Scott Marcus wrote a good comparison of CPP vs Bill and Keep and pointed to the larger amount of minutes of use in BAK countries.


One of the problems with the comparisons between BAK and CPP however is that they are often written in a black and white fashion. The main argument against BAK is that you would have to pay for incoming calls, that no user would/should accept this. Often the argument is used that it is bad for low usage and pre-paid customers as in CPP the operator gets income from their incoming calls too. What isn't done however is to make a comparison between countries with a high termination rate and a low termination rate in CPP countries. It is here that you can see some very interesting developments.

India is the ultimate country when it comes to low termination rates. They are at 20 paise per minute or $0.004/minute. Idea Cellular there reports around 400 minutes of use and an ARPU of $5.70. Interestingly they see a decline in average minutes of use, because of more multi-SIM customers. So an average customer may use more minutes, but on various SIM's. Almost all of these customers are on pre-paid subscriptions. Now contrast this to some other countries in the world.

Telenor in it's quarterly reports gives a good insight into the minutes of use for all of it's countries.
Pakistan has termination rates set at $0.013 per minute.  For Pakistan they give 150 minutes of use on average.

Thailand's termination rates are at around $0.02 per minute. In Thailand there are on average 300 minutes of use.
In Bangladesh the termination rate is at $0.0026 per minute and the Grameen Phone customers call 300 minutes of use.
All three countries aren't rich, but have mobile penetration rates of 30% or more and compound average growth rates of over 50 %, which means that with 2 years they will be above 60% market penetration. 

Now compare this to Telenor's European operations, neither Norway, Denmark or Sweden reach more than 250 minutes of use. Serbia gets 100 minutes of use. What are the termination rates in these countries?
Norway - between $0.08 and $0.14 and 250 mins of use
Denmark - between $0.10 and $0.14 and 200 mins of use
Sweden - $0.06 and 220 mins of use

So the poorer countries in this example have more minutes of use and a lower termination rate. Given the CAGR in the poor countries it can't be said that they don't see an uptake of mobile phone usage. Despite even the fact that in these countries most people use pre-paid mobiles.  All in all to me this implies that there is much to say in favour of decreasing mobile termination rates and maybe to abolish them all together. Also I would expect that developing nations would be in favour of such a move as it becomes clear that the nations with the lowest termination charge are the one's with the highes minutes of use.

Now even RIM says Etisalat was hacking Blackberries

As reported last week, Etisalat installed software to intercept communications from Blackberries on their customers Blackberries. The software, called Interceptor, was developed by SS8 and was of rather shoddy quality, leading to decreased battery times and slower Blackberries. Etisalat denied any wrong doing and claimed it was an upgrade that would improve 2G to 3G handover. Many people, including me, called this utterly bogus.

Now an unsuspected source of support has come. RIM itself has developed and released a security update to remove the SS8 Interceptor application. Here is their announcement:


Registration App Remover for BlackBerry smartphones

Recently an update may have been provided to you by Etisalat for your BlackBerry Handheld via a WAP push. The Etisalat update is not a RIM-authorized update and was not developed by RIM. Independent sources have concluded that the Etisalat update is not designed to improve performance of your BlackBerry Handheld, but rather to send received messages back to a central server. RIM has developed this software (“Software”) that will enable you to remove the Etisalat update.

Etisalat and SS8 hacking your Blackberry for (un) lawful interception

It seems the UAE had some trouble reading Blackberry communications and turned to SS8 for a solution. SS8 suggested an unobtrusive program to be loaded on all Etisalat's customers Blackberry's. 'Trust me guv, nobody will notice'... yeah right. The programme eats batteries for lunch and the server it needed to communicate back with was overloaded (IDIOTS, like you don't know how many devices there are!). Annoyed Blackberry users saw their devices slow down to a crawl and started to complain. A little investigation later and a programmer found out the so called performance upgrade rolled out to all Blackberry users was a snooping programme by the SS8 company.

As some of you might know I've worked on the data retention and lawful interception in the past. I was at the Ministry of Economic Affairs, writing obligations into the Dutch Telecommunications law (chapter 13). The rules of lawful interception are that:

1. you perform it in a way that the target cannot identify whether he/she is being intercepted.
2. the intercepted data is transferred to the government as is and with precautions against tampering with the data on the side of the telecommunications company and on the side of the law enforcement agency.
3. all the communications of the target are intercepted, regardless of the service/channel used (so GSM, GPRS, UMTS etc)

The trouble is that a target may irritate a law enforcement agency by using encrypted communications. So when you intercept that according to the rules, you get intercepted communication that you can't read. (but still can perform cool traffic analysis on, however that is for a different post.) In the past we've seen complaints about Skype and the German/Dutch Cryptophone. Blackberry is also one of the naughty boys who seem to have encryption in place. Not a very good one as the NSA doesn't like it for Obama and from what I can find on the net, the Indians claim to have cracked it.

For the UAE it seems to have been too difficult to do cryptanalysis on the Blackberry. They must have asked around the world of lawful interception vendors and found one who was able to sell them a 'solution'. From personal experience I can tell you that the world of lawful interception vendors is full with dodgy, shady, snake oil vendors. If ever you want to see some of them, go to the ISS world conference. It is way fun as you can read in this Wired story :-) And well SS8 is one of the more respectable ones in this dodgy world. But they're still out to make a quick buck and now have been found with their pants down. This is not the way to do proper interception and even if the technology would have worked as advertised chances are that some Blackberry developer would have figured this one out within no time and would have spilled the beans on-line. What programmer doesn't want to know how a proprietary performance update works.

Now the UAE is no democracy and it doesn't care much for 'rights' as demonstrated in this video, so it doesn't care about due process, but it does care about 'face' and I bet SS8 will have quite some trouble to try and save it's masters face ..... or face a 'similar' punishment as the grain salesman in the video. (who wants to bet that SS8 will not be at ISS World Dubai in 2010?)

Update: I did some searching around and came across this brilliant posting on the Blackberry boards. To really make everything very clear the files in the update were in a directory named: ss8/interceptor.... yeah right... like we don't have Google these days.

UPC Fiber Power rips off Hong Kong FTTH Commercial (badly)

The orginal commercial is funny. It has farts in it.



Here the UPC-commercial, no farts, not funny



For those people who want to know which commercials UPC's creative team will rip off in the future?

http://www.youtube.com/user/HKBNatUTube

for all Hongkong city telecom commercials.

Fiber to my Home, the installation in home

Today fiber was installed in my home. You've seen the previous installments how fiber was just outside my front door. Today the last meter was bridged. This was done by a company called Schuuring. They apparantely are looking for more people to help out.  Two people came. One made sure that the fiber got from the front door to the utilities closet. The second guy did the final work, make the fiber ready for use.

The first guy
He started with opening the latch to the area between the floor and the ground, there is some space there to crouch. After this he opened up the stones in the street and dug a small hole to get the slack fiber from under the pavement. 

From Fibre to my Home

There is about 10m slack available

From Fibre to my Home

A hole is drilled in the outside wall to bring the fiber in. The fiber is run through and the hole is filled with some kind of elastic glue.

From Fibre to my Home

From Fibre to my Home

From Fibre to my Home

The slack is fed into the crouching space and left there. The last two meter is pushed upwards into the utilities closet.

From Fibre to my Home

The first guy is then almost finished. He cleans up after himself, hammers the stones back in. Tells the second guy that he has chipped a stone and promises that it will be replaced. His final task is to leave a plastic bag with a connection set for the second guy. Good thing to know. If the first guy doesn't hammer the stones back in properly and I would complain later on, than he has to come back on his own dime and fix the problem. So he was committed to hammering the stones back into the pavement the right way. (In 5 years time those stones will hover above the rest of the pavement I expect :-) )All in all this took about 15-20 minutes.

From Fibre to my Home

From Fibre to my Home

The second guy
The second guy came about half an hour later. He told me that they had been told not to allow pictures as some of the stuff was proprietary and "copyrighted". He was a young guy and I don't want to bring him into trouble. It seems Arcadis checks up on their work and well. Arcadis developed an online model for the Ministery of Economic Affairs which you can use to calculate the costs of installing a fiber network in The Netherlands. I used it for the calculations in my OECD paper.

I did get some pictures of his kit and the way the fiber was installed. The install itself takes less then 15 minutes and is mostly aimed at getting various layers of the fibers and then connecting the fiber to the connectors (which come with fiber attached). That then is put into the metering closet in the grey cabinet.

From Fibre to my Home

From Fibre to my Home

The splicer! Very cool with a small LCD screen.

From Fibre to my Home

The whole shebang installed in my utility closet. Next to it you can see my Alice DSL modem.

From Fibre to my Home

From Fibre to my Home

The Finished product.

The guy gave me some interesting information as well. An installer is required to do 11 installs per day. Homes like mine are easy. and they can do 20 of them on a good day. High Rise is difficult and takes more time. Older homes are also a pain. they sometimes need to drill straight from outside into the living room. A max of 6db signal loss is allowed. They try to do alot less, between 1 and 3. All in all the installation took only 15 minutes.

The line also needed to be tested. For this he used a unit by Kingfisher, pretty standard. What took the most time was getting someone on the phone at the central office. Those guys need to connect about 200 homes per day and they can be quite busy. There are 2 guys per central office. If someone finds a way in which the central office testing can be automated, that would be a great benefit. In my opinion. If you can get a testing unit to which you can hook up 500 to a thousand homes and have it communicate with the testing unit (or wirelessly to the guys mobile phone) that would be a major improvement to their workflow. Anything around 5 to 10 thousand euro would probably be worth the investment.

All in all this bit took another 15 minutes. 10 minutes for the waiting 5 for the testing.

Now I need to wait for KPN or Online to come up with a symmetrical offer for broadband to my home. Or shall I punish KPN and go for UPC for one year??

Bonus pictures
A picture of how Reggefiber promotes that people have signed up for fiber (glasvezel) in Zeewolde.

From Fibre to my Home

And some pictures of someone sitting in a tent in the street hooking up the fibers from individual homes to the main duct.

From Fibre to my Home

From Fibre to my Home

From Fibre to my Home

From Fibre to my Home

Dilemma: Do I choose Reggefiber/KPN or UPC

Maybe you readers can react and give me a clue of what I should do, because I am in a difficult spot here.
Tomorrow Reggefiber will install the FTTH in my home. Pictures will follow. So in a month or two, I will be able to get KPN over fiber here. Now you would imagine me to go for the fiber offer. I have promoted it everywhere. I still think that fiber is the coolest thing to happen in a long while... However this is the choice I am facing.

I currently spend 80 euro/month on Triple Play. Alice Internet 20/1 DSL+phone international 50 and UPC Royal TV.30
The TV is non-negotiable. BBC 1, 2, 3, 4 are watched alot here. KPN can't offer it yet.

KPN offers me 30/3 internet and free national calls at 55. So with UPC Royaal TV at 30 that makes 85.
UPC offers me 60/6 internet, free national and europe wide calls and UPC Royaal TV at 74 euro/month
UPC even offers me 90/6 internet, free national and europe wide calls and UPC Royaal TV at 84 euro/month

So what should I choose? Go for a year of Docsis 3.0 and hope KPN gets the point or go with fiber even though it misses the sparkles.

Please give a reaction in the comments

Price war in The Netherlands. UPC drops prices

Well, UPC has declared war on KPN. It has drastically dropped its's prices as of today. As you might remember I named UPC as the price quality winner recently. Now it's for a similar offer also the cheaper option. The worst comparison is the 60/6 offer. KPN asks 110 euro for a 60/6 triple play offer. UPC asks 74 euro for a similar offer.  KPN offers free viewing of the Dutch football league, but UPC throws in free calling in Europe (including Turkey)

Based on head-line speeds, KPN has a problem both in the fibre space as in the DSL-space. The public won't understand an argument based on the difference between Ethernet and Docsis 3.0 when it comes to fibre. The headline speeds of ADSL2+ are lower and the price difference is not substantial to UPC's 30mbit offer. For instance Telfort, KPN's cheap brand asks 34,95 for 20mbit and unlimited national calling. UPC asks 48, but throws in a free digital tv and a PVR for those customers that already have analogue Cable TV with UPC (almost 90% of the potential subs in UPC's region)