Facebook adopts LISP to roll out IPv6 Service

This is a diagram drawn by me about how Facebook rolls out its IPv6 service.

LISP (Locater/Identifier Separation Protocol) is amazing. It redefines the relationship between end-point, IP address and router. The IP address assigned to an end-point is simply an identifier and the router is a locator. If the Locator can be spilt into egress and ingress network elements and with the addition of one to one single v4-v6 mapping, then all existing v4 platforms can server v6. Facebook can then save hardware cost, avoid v4 and v6 software clash and most importantly achieve quick roll out of IPv6 service.

There are two questions remaining. First can LISP handle huge traffic and second how can resilience be built into LISP.

No matter how, LISP is a promising new technology which content providers should pay attention to.

IPv6 Cache Servers + IPv4 Forwarders

Our company has implemented an native IPv6 link and we now have an IPv6 only network. However, on testing through test-ipv6.com, the score for IPv6 stability and readiness was just 7/10.  The problem is that the two DNS resolvers riding on IPv6 offered by the serving ISP are just caching servers which pass queries to two forwarders.  The forwarders can only support IPv4.  We are surprised to note that an IPv6 capable ISP does not offer full IPv6 connectivity to its name resolving systems.  This will affect our global IPv6 reachability.

As a small customer, we do not have the power to ask the ISP to change its network architecture.  We can only just alert  the ISP hoping that it will do something positive.  On the  other hand, the cost of setting up our own IPv6 or dual-stack resolvers is insignificant. 

A message to all my IT friends in Facebook

In view of the recent Internet Learning Support Program incident and the advantages given to iProA, I have no choice but to send out a message to all my IT friends in Facebook which reads as follows :


"Dear XXX,


You are in the IT field. Are you a member of iProA ? If so, I am not your friend anymore. I ask this because most of my friends are in the IT Sector. Some of them turn to DAB and iProA to get social and political advantages. I want to make sure all my IT friends are clean from DAB poison. 

Sorry to bother you and ask you."


All replies so far are positive.

OpenDNS offers IPv6 resolvers

Some years ago, I tried OpenDNS resolvers at 208.67.222.222 and 208.67.220.220. The performance is  OK and the two resolvers are on different network to avoid a single point of failure. The good news today is OpenDNS can now provide IPv6 resolvers at 2620:0:ccc::2 and 2620:0:ccd::2 for the Internet community. Looking at the address syntax, the two resolvers are on different /48 subnet and again giving some resilience.

I definitely have the need to use IPv6 resolvers for network configurations, testing and troubleshooting. A big hand to OpenDNS.

Impact of World IPv6 Day to Hong Kong

As World IPv6 Day (W6D) is approaching, IT people in this city start to think how many users will not be able to access Facebook, Google and Yahoo and what is the overall impact to Hong Kong.  If we look at the nature of IPv6 brokenness, it is the behavior of dual-stack clients wrongly select the 6to4 tunnels instead of the native IPv4 path to reach a  destination website that is running both native IPv4 and IPv6.  Hong Kong is quite lucky as Hurricane Electric (HE) has provided a 6to4 gateway with ample bandwidth.  The clients will use 6to4 tunnels to reach HE’s 6to4 gateway and then access Facebook, Yahoo or Google on IPv6 without break.  This works on the condition the serving ISPs do not block protocol 41 in their firewalls and access to the anycast addresss 192.88.99.1 is also not restricted.  Specifically, any intentional blocking of the anycast network 192.88.99.0/24 in the ISP side should not exist.  In other words, the impact to Hong Kong is quite minimal if ISPs are willing to let protocol 41 to pass through.

On checking the number of ASNs advertising 192.88.99.0/24, I notice that there are 33 6to4 gateways in the world. In some other countries where there is no 6to4 gateway, users will experience brokenness on W6D.  

In case of complete breakdown of HE’s 6to4 gateway in Hong Kong, there will be about 2500 users with broken access.  The figure is based on 0.05 % brokenness estimated by ISOC multiplied by 5 million PC users in Hong Kong.  I must say this is an unfounded worry. 

How about congestion in 6to4 gateway.  This should not be a problem since HE’s 6to4 gateway has a bandwidth of 1 Gbps and if 2500 users access the gateway at the same time, each user can have 400 kbps connection speed.

I think I am the first IT people to analyze the impact of W6D to Hong Kong.  I hope my analysis is sound and justifiable. 

Empty a file

I note that most people use /dev/null to empty an existing  file such as :

#cat /dev/null > dnssec.log

Another common usage is "echo -n > dnssec.log" .

My way of clearing all content of a file is odd as I am using tail :

#tail dnssec.log > dnssec.log

Interesting, I forget how, where and when I learnt this crazy command.  Indeed, I don't quite understand  how printing some last lines of a file on screen and then pipe to the file itself can actually clear all the content.

If it is not CISSP, it may not be the best fit

CISSP holders, when accessing ISC2 website (http://www.isc2.org/), please don't be annoyed by the banner of a dog trying to get into a tiny wooden hut. ISC2 is just promoting the status of CISSP with a key message “If it is not CISSP, it may not be the best fit”.

Facebook Internet email addresses are easy to harvest

Facebook offers me the Internet email address as “warren.kwok@facebook.com” as I have a URL of www.facebook.com/warren.kwok for other facebook users to view my profile. I am quite worried that my Facebook email address can be harvested easily and this account will receive large amount of spam emails. I have tried randomly to play with the following URLs after logging in to find if such users in facebook are valid:

www.facebook.com/peter.chan
www.facebook.com/andy.chan
www.facebook.com/david.lee

All are success. That means, I have harvested three valid email addresses @facebook.com.

Without knowing what facebook can do in anti-spam, it gives me no choice but to stop the email account @facebook.com  by setting the privacy preferences as not to receive any Internet email messages.  The return error message below verified that things work up to my expectation:

***** Quote *****
Final-Recipient: rfc822; warren.kwok@facebook.com
Diagnostic-Code: smtp; 550 5.1.1 RCP-P2 http://postmaster.facebook.com/response_codes?ip=202.81.252.116#rcp Refused due to recipient preferences
Action: failed
Last-Attempt-Date: Sun, 22 May 2011 18:37:56 -0700
Status: 5.1.1
***** End of quote *****

Control-Enter shortcut key in IE Browser

When using Chrome and Firefox, if I just type "cnn" followed by CTRL-ENTER, the address bar will make up the site "ww.cnn.com" and the content can be displayed successfully. This is a special hotkey in browser and I think that might be a reason corporation and companies like ".com" so much due to its convenience over other Top Level Domains in browsers.

But what happen to my IE. If I do the same on IE, the address bar will become "www.cnn.com.tw". Hey, the crazy thing is ".tw". It is because my IE browser is a Taiwanese version. I don’t understand why Microsoft is so crazy to insert ".tw" when dealing with the most common CTRL-ENTER shortcut when the IE browser is a Taiwanese version. I am not going to change my IE to English version just because of this crazy flaw. I must say I hate the foolish mindset of Microsoft software people.

chksig - DNSSEC tool for Windows

Right now, there is not any GUI DNSSEC testing tools for Windows other than dig which is command line based. Chksig (http://www.simpledns.com/outbox/chksig.zip)  can be handy to troubleshoot faults in DNSSEC-signed name records in authoritative name servers. 



T



















This tool is bundled with another copy which work on DOS command-line interface.  Using this tool in both the GUI mode and command-line mode are interesting.

nslookup should be phased out

Shit ! A large number of system administrators are still using “nslookup” to test and troubleshoot faults in resolvers and name servers. They should be aware that “nslookup” is an outdated primitive tool which can not offer much help. They should use “dig”.  Dig for Windows is widely avaialable.  Alternatively, they can install BIND for Windows but just use dig without caring to set up an authoritative name server or resolver.

Just ask yourself a simple question, can nslookup tell if a resolver has successfully verified the signature of a queried name record if the zone being interrogated is DNSSEC-signed.

103/8

For those who have visited APNIC after 15 April, they should have noticed a  flashing banner with big words of "103/8" which I have captured below:

103/8 is the final /8 block and when allocation of 103 prefix is started, APNIC has already activated the final /8 policy.   The final /8 policy only allows existing or new members to get /22 (1024) IPv4 addresses which can only be used to build v4<->v6 transition systems for supporting IPv6 networks to reach IPv4 networks.  This banner might stay in APNIC website for up to 5 years.  Its purpose is to remind ISPs and corporations that they should move to IPv6 by now. 

The consumption of v4 addresses in AP Region is alarming.  On Feb 2011, IANA allocated the 39/8 and 106/8 to APNIC.  These two blocks were depleted in early April 2011 prompting APNIC to activate the final /8 policy for the last block.

SOA Expire in Name Servers

Yesterday, all IT people in our department were very angry. We found that our seven domains hosted in the name servers of the ISP were having SOA Expire set as 3600. A dump is below :

The setting of SOA Expire in the name server was problematic. If secondary servers can not contact a primary server due to network outages or whatever reasons, and after 3600 seconds, the information contained in the secondary servers is considered no longer authoritative. Once SOA expiry is reached, the secondary servers will not respond to any query. IETF suggests a  minimum of 1 week and the maximum is 4 weeks.

This was a big mistake. There are cases of power interruption, landslides and cable cut in which the damages last for several hours. Though the serving ISP can claim itself very technically capable to restore server problem within 1 hour, the cases of power lines breaking down, landslides and cable damages due to road digging are outside the control of the serving ISPs. We therefore have to bear the risk  of people not able to reach our various websites due to the ISP’s secondary nameservers not responding which could be other source of failure outside the ISP’s control for more than 1 hour.

We escalated our complaint to the highest level and the problem was rectified. We also learnt a lesson. In future, in case of moves and changes in domain name records, apart from checking the changes are carried out, we must  check the SOA serial numbers which reflect the changes made on a certain day and the SOA Expire is not inadvertently amended.

6in4 or 6to4 tunnel, that's the question

I have been invited by the Internet Society Hong Kong (ISOC-HK) as a guest speaker for the "Kickstart IPv6! Seminar on World IPv6 Day (8 Jun)” which will be held in Cyberport on World IPv6 Day. I will share about my experience of deploying IPv6 in our department, from interim tunneling arrangement to native IPv6 connection. Hey, I don’t mind telling the audiences that I made a big mistake in selecting solution for our interim IPv6 web server. In Jan 2010, I had to decided to use 6in4 or 6to4 tunnel for the web server. 6in4 is offered free by a tunnel broker service provider. It requires login account name and password to set up the tunnel and on Windows server, there must be a start up script to fire up the v6 interface whenever bootup. 6to4 is easier and everything is automatic. If anyone has to choose between 6in4 and 6to4, 6to4 is definitely attractive. One thing that I hate is that Mircrosoft uses a risky 6to4 address format which maps the IPv4 address 202.81.93.74 to 2002:ca51:5d4a::ca51:5d4a (202 dec= ca hex). The 3rd – 6th octets matching 13th – 16th octets tells people that we are using Microsoft OS and hackers can then initiate attacks targeted at Windows OS and IIS. We had to change the 13th – 16th octets to other arbitrary hexadecimal number like 2002:ca51:5d4a::aaaa:ffff.

Everything seemed ok and it was a pretty smart choice at a first glance. Eventually, I discovered that I was wrong to use 6to4 address. It is because web server with 6to4 address can not attract traffic from dual-stack hosts with native IPv6 connection. Over 95 % of visiting addresses were 6to4 addresses and obviously they were Windows 7 hosts running PPPoE or Metro-Ethernet without NAT and their 6to4 tunnels were automatically set up. In that scenario, 6to4 hosts will visit 6to4 servers as they are on the same 6to4 network. However, dual-stack hosts will only use native IPv4 if a web server has native IPv4 address and 6to4 address. 6to4 path will be abandoned while native IPv4 path is selected because 6to4 is less reliable than native 6to4. I applaud that Microsoft do the right thing.

The addresses of 6in4 come from a native IPv6 service providers. All OSes can not tell if these addresses are native IPv6 or 6in4. Hence, if we had use 6in4 in the first place, our interim web server should have attracted a high traffic. Though we discovered that weakness, we did not want to switch back to 6in4 as we will use native IPv6 connection very soon.

This is a painful experience. I hope other network administrators will take my advice to use 6in4 tunnel as opposed to 6to4 tunnel when considering interim IPv6 solution.

Equivalent of link-local address fe80::/64 in IPv4

Some days ago, my friends asked me what is the equivalent of IPv6 link-local address fe80::/64 in  IPv4.  Link-local address is auto-configured in the absence of router or DHCPv6 server.  In IPv4, we have 169.254.0.0/16 random addresses assigned to a NIC if it can not find any DHCP server.  Most importantly, hosts with 169.254.X.Y can communicate with each other since they are in the same /16 subnet.

However, there should be one difference, only Windows and Mac stations support 169.254.0.0/16.  While in IPv6, all OSes must support the link-local address and this should include iPhone OS and Android.

Server up for 600 days without reboot

My server running Fedora Core has been up and running for 600 days without reboot. This is amazing.  Can anyone expect the same for Windows 2003 and Windows 2008 Server.  I also need to thank the data center for providing a very stable power supply.  Obviously, there has not been any power interruption.

During the past 600 days, I applied patches and updates to some software packages.  The beauty of Linux is that there is no need to reboot server when new patcches are applied and new software packages are installed.

Caching period of NXDOMAIN

I have the wrong idea that only positive answers of name lookup will be cached with a period defined by the TTL while negative answers (NXDOMAIN) will not have the same function.  The fact is NXDOMAIN will also be cached in a resolver and the period is according to the “SOA Minimum” of the zone file.  Suppose if I interrogate xyz.cnn.com at a resolver, the name server will reply NXDOMAIN and the provide the SOA record which contains origin, mail address, serial, refresh, retry, expire and minimum. For this case, the SOA minimum is 3600.  If after 10 seconds, I ask xyz.cnn.com, the resolver will fetch the answer from the cache indicating the remaining time is 3590. 

Funs. This tells why hackers can inject fake NXDOMAIN to make a domain name inaccessible as a way of DOS attack.    

Windows 7 Router Advertisement DOS attack

The youtube video below demonstrates that rogue Router Advertizements can be a serious DOS attack, which can crash all IPv6 Windows systems in a local area network.

http://www.youtube.com/watch?v=GA_w87K_Iuo&feature=player_embedded

This has been reported to Microsoft for some time. As usual, Microsoft says there is is no fix at this moment.

This scenario will not happen to in our office as we know for sure autoconfiguration presents too many risks in a corporate network environment. We will use DHCPv6 for v6 address allocation and management. We can therefore turn off router discovery function in all Windows 7 machines.

Estimation of IPv6 brokenness

Eric Vyncke has been gathering live data for the estimation of  IPv6 broknenness  (http://vyncke.org/testv6/) since Oct 2010. As of today, his estimation derived from the gathered data reveals that there might be up to 0.7 % IPv6 brokenness. However, Yahoo estimates that the number of users affected by broken IPv6 is about 0.05 % of global Internet population (2 billion at present) or equal to 1  million. Yahoo's estimation is just  1/14 of Eric Vyncke's test data.



            (Please click to enlarge)

Which one should I believe more, why and how ???

If it turns out that IPv6 brokenness can be up to 0.7 %, then there is no chance Facebook, Yahoo and Google will activate their content on IPv6.

Anyway, the IPv6 World Day 24-hour test flight might provide some useful clues. I am eager to hear the announcement of  Facebook, Google and Yahoo after they have analyzed their  captured data on IPv6 World Day.

IPv6 speed test again

I mentioned about IPv6 speed test in my blog post yesterday.  To my surprise, the post was read by a visitor and he left comments to me that there is another speed test engine in Sweden, URL as http://ipv6.bredbandskollen.se which I have tested successfully.

Well, that's the power of blogging.  Through the blogosphere, people can share knowledge and experience and help each other.

IPv6 speed test

Up to now, I can only find one single working website which offers IPv6 speedtest. 

The test site is in France.  If I use the 6in4 tunnels offered by Hurricane Electric, the speed can be over 5 Mbps.  It is because Hurricane Electric has established IPv6 POP in France which has a fat pipe linking up with the POP in Hong Kong.

Another website is ipv6-speedtest.net.  However, I fail to perform succesful test with this site.

Using email address for login to some online services

Many service providers require users to use their email addresses to login. I like to say that when authenticating a user, there is no need to care the small or capital letters in domain part of  the email address.  In fact, there is no small or capital letter differentiation in a Fully Qualified Domain Name. However, some service providers impose restriction that the small and capital letters of the domain part must exactly matches the one when the user first created the account.

As an example, I have an account with isc.org with the login user as warren@i3way.net.  If I purposely type the user as warren@I3WAY.NET, it says the account does not exist. 

Facebook has considered this aspect carefully.  Equally I have the login user as warren@i3way.net.  If I type warren@I3WAY.NET. Facebook recognizes it as the same account as warren@i3way.net and it allows me to access.

The over-restriction mentioned above is not logical and not sensible if one considers the domain name convention.  For the programmers, it just involves a few more lines of codes which could make things work better.

Number of 6to4 gateway in the globe

I got a list of 33 ASNs that advertize the anycast Class C range 192.88.99.0/24 in their BGP routes:


AS59, AS559, AS680, AS1103, AS1239, AS1257, AS1299, AS1835, AS1930, AS2018, AS2116, AS4621, AS6939, AS7575, AS8359, AS8473, AS12573, AS12779, AS12871, AS15598, AS16150, AS19255, AS19782, AS20312, AS20640, AS25192, AS28917, AS35244, AS38646, AS39326, AS39556, AS44581, AS55374

Some people just wonder why I am interested in tracking the number of ASNs that handle 192.88.99.0/24.  It is because the anycast address 192.88.99.1 is assigned to 6to4 relay.  In other words, there are now 33 service providers that offer free 6to4 gateway service.  Well, this is a thriving development.  They are helping the transition to IPv6 for clients using 6to4 tunnels.

Well-done !

Websites for testing if a resolver is DNSSEC-enabled

Normally, to test if a resolver is DNSSEC-enabled or not, IT people will  use "dig +dnssec"  followed by something and then verify that the message has the AD (Authenticated Data) set.  Instead of using such command line testing,  I am a bit lucky to find two webistes for performing the same task:

1.  http://dnssec-or-not.net/

2.  http://dnssectest.sidn.nl/

The screen dumps below are interesting.

Best view for websites

I found some websites still have the following footnote at the bottom of  a page:

"This site is best viewed at 1024x768 screen resolution with Internet Explorer 6.0 SP2 or above."

The above footnote is absolutely not necessary. Today, people are using 19 - 22 inches LCD with 16:9 aspect ratio.  Some larger monitors have 22:10 ratio.   1024 x 768 is considered as a poor resolution using the old traditional  4:3 aspect ratio.

Just take away the fucking advice and users will know how to adjust their monitors and video display to achieve the best viewing effect.

Congested NATed hosts

This is a funny picture.  It can be used to describe the congested scenario of NATed hosts in IPv4 Internet. The single train is the one and only one routable IPv4 address.  The linited seats (overall capacity) inside the train correponds to port numbers.   This traing will clash and no doubt you are killing yourself if continue to use NAT.

IPv6 Anti-spam nightmare

People are worried that IPv6 might create an anti-spam nightmare because IP-based blacklist systems will not be effective.  In IPv6 environment,  in a subnet, there are 2^64 IPv6 addresses to use.  Spammers can change their IPv6 addresses every single second or send out each spam mail with a different address.  With this uncertainty, some even say that there is no need to rush to implement IPv6 SMTP server as long as they have IPv4 SMTP servers to serve their corporate email communications.

This is a short-sighted view.  There are at least 2 methods I can think to circumvent IPv6 spamming hosts  which can change addresses frequently:

1.  Mandatory reverse lookup - unlike legitimate mail server, the random changing addresses of spamming hosts will not have proper reverse lookup records,

2.  Greylisting - spamming hosts will not queue up mail for subsequent retry but legitimate email servers perform this function.

Of course, above all, there is the solution of content filtering.  This method is more expensive as it requires the use of more CPU power, memory and a monthly subscription fee to obtain the database of spam signatures.