Configure Apache as a caching proxy fronting these services. This means that you can tolerate downtime for the services and you have quicker builds (since you don’t need to contact remote servers). It also has a security benefit (you can firewall of your build server such that it can’t make any outgoing connections) and it’s nice to avoid consuming the bandwidth of those registries (especially since they are provided for free).

Without further ado, here are the config bits for Apache 2.4

/etc/apache2/force_cache_proxy.conf – the general configuration file for caching:

# Security - we don't want to act as a proxy to arbitrary hosts
ProxyRequests Off
SSLProxyEngine On
 
# Cache files to disk
CacheEnable disk /
CacheMinFileSize 0
# cache up to 100MB
CacheMaxFileSize 104857600
# Expire cache in one day
CacheMinExpire 86400
CacheDefaultExpire 86400
# Try really hard to cache requests
CacheIgnoreCacheControl On
CacheIgnoreNoLastMod On
CacheStoreExpired On
CacheStoreNoStore On
CacheStorePrivate On
# If remote can't be reached, reply from cache
CacheStaleOnError On
# Provide information about cache in reply headers
CacheDetailHeader On
CacheHeader On
 
# Only allow requests from localhost
<Location />
        Order Deny,Allow
        Deny from all
        Allow from 127.0.0.1
</Location>
 
<Proxy *>
        # Don't send X-Forwarded-* headers - don't leak local hosts
        # And some servers get confused by them
        ProxyAddHeaders Off
</Proxy>

# Small timeout to avoid blocking the build to long
ProxyTimeout    5

Now with this prepared we can create the individual configurations for the services we wish to proxy:

For pypi:

# pypi mirror
Listen 127.1.1.1:8001

<VirtualHost 127.1.1.1:8001>
        Include force_cache_proxy.conf

        ProxyPass         /  https://pypi.python.org/ status=I
        ProxyPassReverse  /  https://pypi.python.org/
</VirtualHost>

For npm:

# npm mirror
Listen 127.1.1.1:8000

<VirtualHost 127.1.1.1:8000>
        Include force_cache_proxy.conf

        ProxyPass         /  https://registry.npmjs.org/ status=I
        ProxyPassReverse  /  https://registry.npmjs.org/
</VirtualHost>

After configuration you need to enable the site (a2ensite) as well as needed modules (a2enmod – ssl, cache, disk_cache, proxy, proxy_http).

Finally you need to configure your package manager clients to use these endpoints:

For npm you need to edit ~/.npmrc (or use npm config set) and add registry = http://127.1.1.1:8000/

For Python / pip you need to edit ~/.pip/pip.conf (I recommend having download-cache as per Stavros’s post):

[global]
download-cache = ~/.cache/pip/
index-url = http://127.1.1.1:8001/simple/

If you use setuptools (why!? just stop and use pip :-)), your config is ~/.pydistutils.cfg:

[easy_install]
index_url = http://127.1.1.1:8001/simple/

Also, if you use buildout, the needed config adjustment in buildout.cfg is:

[buildout]
index = http://127.1.1.1:8001/simple/

This is mostly it. If your client is using any kind of local caching, you should clear your cache and reinstall all the dependencies to ensure that Apache has them cached on the disk. There are also dedicated solutions for caching the repositories (for example devpi for python and npm-lazy-mirror for node), however I found them somewhat unreliable and with Apache you have a uniform solution which already has things like startup / supervision implemented and which is familiar to most sysadmins.

Here are the numbers I got. This is on a Core 2 Duo T5500 @ 1.66 Ghz processor. The numbers express Mbits/sec processed:

• The assembly version from the blogpost (table taken from here): ~1700
• Optimized C implementation (taken from the same source): ~1500. The compiler used was Microsoft Visual C++ Express 2010
• Unoptimized C implementation (ie. Debug build): ~900
• Java implementation using polynomials: ~100 (using JRE 1.6.0_23)
• Java implementation using table: ~1900
• Built-in Java implementation: ~1700
• Javascript (for the fun of it) implementation (using the code from here with optimization – storing the table as numeric rather than string) on Firefox 4.0 Beta 10: ~80
• Javascript on Chrome 10.0.648.18: ~40
• (No IE9 test – they don’t offer it for Windows XP)

Final thoughts:

• Hand coding assembly is not necessary in 99.999% (then again 80% of all statistics are made up :-p). Using better tools or better algorithms (see the “Java table based” vs. “Java polynomial”) can give just as good of performance improvement. Maintainability and portability (almost always) trump performance
• Be pragmatic. Are you sure that your performance is CPU bound? If you are calculating a CRC32 of disk files, a gigabit per second is more than enough
• Revisit your assumptions periodically (especially if you are dealing with legacy code). The performance characteristics of modern systems (CPUs) differ enormously from the old ones. I would wager that on an old CPU with little cache the polynomial version would have performed much better, but now that we have CPU caches measured in MB rather than KB the table one performs much better
• Javascript engines are getting better and better.

Some other interesting remarks:

• The source code can be found in my repo. Unfortunately I can’t include the C version since I managed to delete it by mistake
• The file used to benchmark the different implementations was a PDF copy of the Producing Open Source Software book
• The HTML5 implementation is surprisingly inconsistent between Firefox and Chrome, so I needed to add the following line to keep them both happy: var blob = file.slice ? file.slice(start, len) : file;
• The Javascript code doesn’t work unless it is loaded via the http(s) protocol. Loading it from a local file gives “Error no. 4”, so I used a small python webserver
• Javascript timing has some issues, but my task took longer than 15ms, so I got stable measurements
• The original post mentions a variation of the algorithm which can take 16 bits at one (rather than 8) which could result in a speed improvement (and maybe it can be extended to 32 bits)
• Be aware of the “free” tools from Microsoft! This article would have been published sooner if it wasn’t for the fact MSVC++ 2010 Express require an online registration and when I had time I had no Internet access!
• Update: If you want to run the experiment with GCC, you might find the following post useful: Intel syntax on GCC

Picture taken from the TheGiantVermin’s photostream with permission.

The problem was that the server hosting the SWF and JS file didn’t serve them anymore, instead giving a 403 – access refused error. To mitigate this problem I’ve uploaded the SWF file to Google Code and used the JS file from the Google Ajax Library and bought the plugin back to life.

So, if you are using the plugin and you are subscribed to my feed, go to the original (now updated) post and use the new code.

Thank you and sorry for any inconvenience caused!

• Customizable (password length, types of characters included, etc)
• Secure (it doesn’t communicate over the network, hence no need for SSL)
• Fully reviewable (as opposed to server-based solutions, where you have to trust the server)

The only flaw it had (as pointed out by a commenter) was the fact that passwords didn’t always include all the characters you’ve selected (ie. the checkboxes represented “possible” not “mandatory” characters, which was a little counter-intuitive).

I’ve thought about how to create passwords which included at least one character from each set. My first ideas were around generating a password, then checking that it contained at least one character from each set and if not, replacing some of the characters with ones from the missing set. However this train of thought quickly ran into problems when I had to decide which character to replace. Choosing something fixed (like the first one, last one, etc) is too predictable. If I choose a random one, I run the risk of overwriting previous change. So finally I realized that there is a simple solution: just re-generate the password until it satisfies all of the constraints. Although this might seem like a brute-force solution, in practice its speed is indistinguishable from a constant-time solution.

Below you have the new and improved YARPG:

I’ve also updated the original posting. You can get the source code for it by looking at the source of this webpage, or from my SVN repository: js_password_generator.html. Hopefully you find it useful!

Picture taken from cjc4454’s photostream with permission.

Of course this is an unofficial and unsupported solution, so it might break at any time and without warning!

YouTube username (for example homestarrunnerdotcom): Show me!

What bothers me:

• The blacklist approach – this means that there will be a lag before new attacks are detected
• Relying on third-party service (like the Google Safe Browsing API, McAfee SiteAdvisor, etc). While the Google Safe Browsing API has an explicit TOS stating that you can use it (under certain circumstances of course), the situation with McAfee and Symantec is not as clear-cut. Does Dasient have a contract with them or are they just scraping their websites? What if McAfee / Symantec decides that enough is enough and blocks them or even worse, sues them? Also, relying on these services means further delay in detecting the infected sites (because they must wait until these providers detect the infection)
• Their touted “dynamic filtering” technology seems to be over engineered for me. It also (as far as I understand) can’t handle situations like “the request is directed to a different machine” or “the machine is rootkitted and the malicious code is added on-the-fly”, both of these being situations which occurred in the real world (the first with CN CERT and the second with a bunch of compromised Linux machines)
• Also, I fear that because this filtering masks the problem (much like a WAF does), it will encourage people to be complacent about fixing the root of the problem (“so what if we get compromised twice a day due to weak passwords? we just click the checkbox!”)
• Finally, the prices seem a little steep to me (starting from ~10 USD a month and going over ~ 50 USD per month)

All in all it doesn’t seem to me to be worth 2M USD (which they claim to have in funding)…

So, if you like my ramblings, please subscribe. If you don’t, leave a comment and subscribe :-).

But today I wanted to praise two widgets which IMHO get it right. I’m talking about Feedjit which shows the (approximate) physical location of your last N visitors and the Weather Sticker from Weather Underground. The things I like in them:

• Minimal fuss (no signup) required to get them.
• They are just an image! (Feedjit offers also a javascript version, but it is optional)
• Given that it is just an image, the browser can download it in parallel and doesn’t need to suspend the processing of the page.

Even though it is just an image, the provider gets a link back. Also, it can

Finally, it works when Javascript is not available and it eliminates a very obvious way to attack your users!

I wish that all gadget providers would offer a lo-fi version of their gadgets. A+ to both of these services (and you can find them in my sidebar).

<script>var start=new Date();</script>

<script src="http://ad.a8.net/foo.js"></script>
<script src="http://asy.a8ww.net/foo.js"></script>
<script src="http://a9rhiwa.cn/foo.js"></script>
<script src="http://www.a9rhiwa.cn/foo.js"></script>
<script src="http://acezip.net/foo.js"></script>

<script>var stop=new Date(); alert(stop.getTime() - start.getTime());</script>

What this code does, is to try to include javascript files from five sites and measure the time it takes to process these tags. This testcase was selected because current browsers load the scripts synchronously (the reasons being that the loaded script might modify the current document) and also it is a commonly used method for including third-party content (advertisements, gadgets, etc). The result are:

• Firefox (3.5) showed no difference: the timing always was a couple of milliseconds
• Internet Explorer (8) and Google Chrome (2.0) showed quite a large difference in favor of 0.0.0.0: when using it, the pages constantly loaded in tens of milliseconds (between 0 and 30), while using 127.0.0.1 meant a page load time closer to a second (between 900 and 1500 milliseconds).
• Using Opera (9.64) the results were even further apart: the 0.0.0.0 case took ~1 second, while the 127.0.0.1 case took around 25 (!!!) seconds.

All the measurements were repeated multiple times to ensure their validity. The machine used for this test was running Windows XP with all the latest updates and without any webserver. IMHO this is yet an other argument in favor of using 0.0.0.0 instead of 127.0.0.1 when trying to block hosts.