Every end is a new beginning

TL;DR: I’m shutting down the twitfeeder project (it was on lifesupport for a long time) so I mirrored a technical article from the blog in the hope that it might be useful for somebody someday.

Proxying URL fetch requests from the Google App Engine

Hosting on the Google App Engine means giving up a lot controls: your
application will run on some machines in one of Google’s datacenters,
use some other machines to store data and use yet an other set of
machines to talk to the outside world (send/receive emails, fetch URLs,
etc). You don’t know the exact identity of these machines (which for
this article means their external IP address), and even though you can
find out some of the details (for example by fetching the ipchicken page
and parsing the result or by sending an e-mail and looking at the
headers), there is no guarantee made that the results are repeatable
(ie. if you do it again and again you will get the same result) in the
short or long run. While one might not care about such details most of
the time, there are some corner cases where it would be nice to have a
predictable and unique source address:

You might
worry that some of the exit points get blocked because other
applications on the Google App Engine have an abusive behavior

You might want a single exit point so that you can “debug” your traffic at a low (network / tcpdump) level

And finally, the main reason for which the Twit Feeder uses it: some third-party services (like Twitter or Delicious) use the source IP to whitelist requests to their API

To
be fair to the GAE Architects: the above considerations don’t affect
the main usecase and are more of a cornercase if we look at the average
application running on the GAE. Also, having so few commitments (ie.
they don’t stipulate things like “all the URL fetches will come from the
10.0.0.1/24 netblock”) means that they are free to move things around
(even between datacenters) to optimize uptime and performance which in
turn benefits the application owners and users.

Back to
our scenario: the solution is to introduce an intermediary which has a
static and well known IP and let it do all the fetching. Ideally I would
have installed Squid and and
be done with it, but the URL fetch service doesn’t have support for
proxies currently. So the solution I came up with looks like this:

Get
a VPS server. I would recommend one which gives you more bandwidth
rather than more CPU / memory / disk. It is also a good idea to get one
in the USA to minimize latency from/to the Google datacenters. I’m
currently using VPSLink and I didn’t had any problems (full disclosure: that is a referral link and you should get 10% off for lifetime if you use it).

Install Apache + PHP on it

Use a simple PHP script to fetch the page encoded in a query parameter using the php_curl extension.

To spice up this blogpost ;-), here is a diagram of the process:

A couple of points:

Taking
care of a VPS can be challenging, especially if you aren’t a Linux
user. However failing to do so can result in it being taken over by
malicious individuals. Consider getting a managed VPS. Also, three quick
security tips: use strong passwords. move the SSH server to a
non-standard port and configure your firewall to be as restrictive as
possible (deny everything, and open ports to only a limited number of IP
addresses).

Yes, this introduces a single point of failure
into your application, so plan accordingly (while your application is
small, this shouldn’t be a problem – as it grows you can get two VPS’s
at two different data centers for example for added reliability).

The
traffic between Google and your VPS can be encrypted using TLS (HTTPS).
The good news is that the URL fetcher service doesn’t check the
certificates, so you can use self-signed ones. The bad news is that the
URL fetcher doesn’t check the certificates, so a determined attacker can
relatively easily man-in-the-middle you (but it protects the data from
the casual sniffer).

Be aware that you need to budget for
double of the amount of traffic you estimate using at the VPS (because
it needs to first download it and then upload it back to Google). The
URL fetcher service does know how to use gzip compression, so if you are
downloading mainly text, you shouldn’t have a bandwidth problem.

PHP
might seem like an unusual choice of language, given how most GAE users
have experience in either Python or Java, but there are a lot of
tutorials out there on how to install it (and on modern Linux
distribution it can be done in under 10 minutes with the help of the
package manager) and it was the one I was most comfortable with as an
Apache module.

Without further ado, here are the relevant sourcecode snippets (which I hereby release into the public domain):

The PHP script:

<?php
error_reporting(0);
require 'privatedata.php';

if ($AUTH != @$_SERVER['HTTP_X_SHARED_SECRET']) {
   header("HTTP/1.0 404 Not Found");
   exit;
}

$url = @$_GET['q'];
if (!isset($url)) {
   header("HTTP/1.0 404 Not Found");
   exit;
}

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_ENCODING, '');
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);

if (isset($_SERVER['HTTP_USER_AGENT'])) {
   curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
}
if (isset($_SERVER['PHP_AUTH_USER']) && isset($_SERVER['PHP_AUTH_PW'])) {
   curl_setopt($ch, CURLOPT_USERPWD, "{$_SERVER['PHP_AUTH_USER']}:{$_SERVER['PHP_AUTH_PW']}");
}

if (count($_POST) > 0) {
   curl_setopt($ch, CURLOPT_POST, true);
   $postfields = array();
   foreach ($_POST as $key => $val) {
       $postfields[] = urlencode($key) . '=' . urlencode($val);
   }
   curl_setopt($ch, CURLOPT_POSTFIELDS, implode('&', $postfields));
}

$headers = array();
function header_callback($ch, $header) {
   global $headers;
   // we add our own content encoding
   // also, the content length might vary because of this
   if (false === stripos($header, 'Content-Encoding: ')
       && false === stripos($header, 'Content-Length: ')
       && false === stripos($header, 'Transfer-Encoding: ')) {
       $headers[] = $header;
   }
   return strlen($header);
}

curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'header_callback');

$output = curl_exec($ch);
if (FALSE === $output) {
   header("HTTP/1.0 500 Server Error");
   print curl_error($ch);
   exit;
}
curl_close($ch);

foreach ($headers as $header) {
   header($header);
}
print $output;

The “privatedata.php” file looks like this:

<?php

$AUTH = '23tqhwfgj2qbherhjerj';

Two
separate files are being used to avoid submitting the password to the
source repository, while still keeping all the sourcecode open.

Now with the code in place, you can test it using curl:

curl --compressed -v -H 'X-Shared-Secret: 23tqhwfgj2qbherhjerj' 'http://127.0.0.1:1483/fetch_url.php?q=http%3A//identi.ca/api/help/test.json

As
you can see, a custom header is used for authentication. An other
security measure is to use a non-standard port. Limiting the requests to
the IPs of the Google datacenter from the firewall would be the ideal
solution, but given that this was the problem we are trying to solve in
the first place (the Google datacenters not having an officially
announced set of IP addresses), this doesn’t seem possible.

Finally, here is the Python code to use the script from withing an application hosted on the GAE:

from urllib import quote
from google.appengine.api import urlfetch

# ...

fetch_headers['X-Shared-Secret'] = '23tqhwfgj2qbherhjerj'
result = urlfetch.fetch(url='http://127.0.0.1:1483/fetch_url.php?q=%s' % quote(url), payload=data,
 method=urlfetch.POST if data else urlfetch.GET,
 headers=fetch_headers, deadline=timeout, follow_redirects=False)

A little more discussion about the code:

The
method of using a custom header for authorization was chosen, since the
forwarding of authentication data (ie. the “Authorization” header) was
needed (specifically this is what the Twitter API uses for verifying
identity)

Speaking of things the script forwards: it does
forward the user agent and any POST data if present. The forwarding of
other headers could be quite easily be added.

Passing
variables in a GET request is also supported (they would be
double-encoded, but that shouldn’t be a concern in but the most extreme
cases)

If we are talking about sensitive data, cURL (and the cURL extension for PHP) has the ability to fetch HTTPS content and to verify the certificates.

While
this method might look cumbersome, in practice I found it to work quite
well. Hopefully this information will help others facing the same
problem. A final note: if you have questions about the code or about
other aspects, post them in the comments. I will try to answer them as
fast as possible. I’m also considering launching a “proxy service” for
GAE apps to make this process much more simpler (abstracting away the
setup and administration of the VPS), so if you would be interested in
paying a couple of bucks for such a service, please contact me either
directly or trough the comments.

Proxying URL fetch requests from the Google App Engine

Leave a Reply Cancel reply