Here is the list of programming related advent calendars for 2013 (if you know of more, leave a comment and I’ll update the list):

• 24 ways – “is the advent calendar for web geeks. Each day throughout December we publish a daily dose of web design and development goodness to bring you all a little Christmas cheer”
• The Perl Advent Calendar 2013
• Perl 6 Advent Calendar (if you didn’t hear – Perl 6 is a completely different language from Perl 5!)
• SysAdvent (not only) for sysadmins
• Performance Calendar Learn all about web performance. (Primarily focused on frontend technologies.)
• UXmas UX is for everyone. Brush up on the topic this December.
• Go Advent – the go(lang) advent calendar
• 24 Pull Requests – “a yearly initiative to encourage developers around the world to send a pull request every day in December up to Xmas”

The mistakes it makes:

• It tries to do the update when I’m trying to start Grooveshark (it interferes with my intention)
• It consumes 100% of a core by polling for the presence of running applications (I suppose), effectively obliging me to do the update. This is combined with frequent releases (which otherwise would be a good thing) for maximum annoyance.
• Although you can’t see it in the screenshot, the updater has (had?) a bug when it asks for your sudo password: if you misstype it at first, then it asks for the root password (which doesn’t exists under Ubuntu by default) and then it just gets into some weird state until the next update is released.

To sum it up: You should download and install the updates in the background (in a separate, versioned directory, always keeping just the two most recent versions). Users shouldn’t be bothered with this, especially when they are trying to get work done!

Here are the numbers I got. This is on a Core 2 Duo T5500 @ 1.66 Ghz processor. The numbers express Mbits/sec processed:

• The assembly version from the blogpost (table taken from here): ~1700
• Optimized C implementation (taken from the same source): ~1500. The compiler used was Microsoft Visual C++ Express 2010
• Unoptimized C implementation (ie. Debug build): ~900
• Java implementation using polynomials: ~100 (using JRE 1.6.0_23)
• Java implementation using table: ~1900
• Built-in Java implementation: ~1700
• Javascript (for the fun of it) implementation (using the code from here with optimization – storing the table as numeric rather than string) on Firefox 4.0 Beta 10: ~80
• Javascript on Chrome 10.0.648.18: ~40
• (No IE9 test – they don’t offer it for Windows XP)

Final thoughts:

• Hand coding assembly is not necessary in 99.999% (then again 80% of all statistics are made up :-p). Using better tools or better algorithms (see the “Java table based” vs. “Java polynomial”) can give just as good of performance improvement. Maintainability and portability (almost always) trump performance
• Be pragmatic. Are you sure that your performance is CPU bound? If you are calculating a CRC32 of disk files, a gigabit per second is more than enough
• Revisit your assumptions periodically (especially if you are dealing with legacy code). The performance characteristics of modern systems (CPUs) differ enormously from the old ones. I would wager that on an old CPU with little cache the polynomial version would have performed much better, but now that we have CPU caches measured in MB rather than KB the table one performs much better
• Javascript engines are getting better and better.

Some other interesting remarks:

• The source code can be found in my repo. Unfortunately I can’t include the C version since I managed to delete it by mistake
• The file used to benchmark the different implementations was a PDF copy of the Producing Open Source Software book
• The HTML5 implementation is surprisingly inconsistent between Firefox and Chrome, so I needed to add the following line to keep them both happy: var blob = file.slice ? file.slice(start, len) : file;
• The Javascript code doesn’t work unless it is loaded via the http(s) protocol. Loading it from a local file gives “Error no. 4”, so I used a small python webserver
• Javascript timing has some issues, but my task took longer than 15ms, so I got stable measurements
• The original post mentions a variation of the algorithm which can take 16 bits at one (rather than 8) which could result in a speed improvement (and maybe it can be extended to 32 bits)
• Be aware of the “free” tools from Microsoft! This article would have been published sooner if it wasn’t for the fact MSVC++ 2010 Express require an online registration and when I had time I had no Internet access!
• Update: If you want to run the experiment with GCC, you might find the following post useful: Intel syntax on GCC

Picture taken from the TheGiantVermin’s photostream with permission.

The downside is that there seems to be a 30% to 40% performance impact based on some quick benchmark I’ve done. However one can not understate the value of not having to write or maintain code!

There are surprisingly many things one can do in Java in a typesafe manner (including things usually associated with dynamic languages) which helps catching more errors early (at compile time) and take full advantage of the different helper features available in IDEs (such as auto-complete).

To illustrate the problem, I’ve created a small program. It does the following:

1. It creates a Date object and sets it to certain values
2. Schedules multiple Runnable’s which examine the value of the object on a threadpool

Everything looks fine and dandy, right? The object isn’t changed (apparently) after being handed of to the threadpool, yet sometimes wrong answers still appear (it takes around ~30 min on my laptop for such an event). So what are the lessons here?

• Get your API right! If the user doesn’t seem to be doing writing, don’t do writing!
• You can still do lazy initialization (if you really want to), but be sure to make it thread-correct (volatile, synchronized, etc) or at least document it (even though nobody reads the documentation)
• Source code FTW! I couldn’t have debugged this without source code. Ok, maybe I could (decompiling class files is not that hard), but probably I wouldn’t have bothered.
• Finally, the solution (hack) in this particular situation is to call getTime() after setting the values, which preemptively normalizes the internal representation. Of course the proper solution would be to pass around truly immutable objects (like timestamps or value objects from Joda Time).

• Javascript
• Perl
• PHP
• Python
• Ruby
• LUA

They are usually used to quickly evaluate a DSL (Domain Specific Language) expression. What I mean by this is the following: lets say that the user supplies an expression which can be easily (ie. with a few string replacements or regular expressions at most) converted into a valid expression in the current language. Then you don’t have to write your own lexer / parser / runtime to support this function.

To make this example even more concrete, lets say that you are implementing a simple graphing calculator where the user can supply the right part of the f(x)=… expression and you draw the function for a given interval of x. If the user supplies something like 1 + 2*x + 3*x*x, this is pretty much a valid expression in all programming languages (there are minor syntactic differences to be precise – like Perl/PHP requiring you to prefix variable names by the “$” sign), so you could simply use “eval” on it.

Warning! Running eval on unverified, user supplied code is a really, really bad idea! (yes, I know that red and bold underline is a little over the top, but this is just that bad! Never, never, ever do this! It is equivalent to letting everybody connected to the Internet (assuming that we are talking about an webapp) running arbitrary code on your server. Implement very strict filtering (based on whitelisting if at all possible) for such features!

Surely, you would say, such a dynamic feature isn’t easily accessible for a statically typed compiled language as Java… And you would be wrong! As of Java 6 each JVM install (including the JREs) includes the Java compiler, and it also includes a public API to access it. Using this feature you can implement the Java equivalent of “eval”: giving a string to the compiler and getting a class instance back, on which you can call methods. You can find the source in my SVN repo. It is (almost entirely) based on the following article from 2007 (just to give you an idea how long this option has been around): Create dynamic applications with javax.tools. An other (pleasant) surprise was the fact that this process doesn’t require any security privileges and works perfectly in restricted environments such as browser.

An additional advantage of using the JVM rather than your own runtime is speed: many man-hours have gone into optimizing both the source –> bytecode and the bytecode –> machine code transformations. Which brings me to an other possible use for this kind of solution: generating particularized instances of generic classes to give more hints to the JVM about possible implementations.

For example, the StrinkTokenizer class does the following when looking for separator characters:

char c = str.charAt(position);
if ((c > maxDelimCodePoint) || (delimiters.indexOf(c) < 0))
    break;

Now imagine how much more efficient (in the sense of: easier for the JVM to translate into an efficient machinecode) this code would be if we knew that we have exactly one possible delimiter (as it is the case most of the time). Replacing delimiters.indexOf(c) with delimiter == c can give you an order of magnitude speedup for this particular code.

The takeaway should be:

• This is a very powerful technique, but it should be used with care! Only use this method if you’ve proven (by using a profiler for example) that the given class is the dominant factor in the performance picture.
• Be particularly aware of potential security risks which could appear!
• Also, be aware that you give up many things when going this route:
  • Automated refactoring
  • Reports generated by bytecode analysis tools (like coverage or bug detection)
  • Debugger support
• In conclusion: use it with great care, but if used properly, it can result in considerable performance improvements!

Picture taken from Hexadecimal Time’s photostream with permission.

• Customizable (password length, types of characters included, etc)
• Secure (it doesn’t communicate over the network, hence no need for SSL)
• Fully reviewable (as opposed to server-based solutions, where you have to trust the server)

The only flaw it had (as pointed out by a commenter) was the fact that passwords didn’t always include all the characters you’ve selected (ie. the checkboxes represented “possible” not “mandatory” characters, which was a little counter-intuitive).

I’ve thought about how to create passwords which included at least one character from each set. My first ideas were around generating a password, then checking that it contained at least one character from each set and if not, replacing some of the characters with ones from the missing set. However this train of thought quickly ran into problems when I had to decide which character to replace. Choosing something fixed (like the first one, last one, etc) is too predictable. If I choose a random one, I run the risk of overwriting previous change. So finally I realized that there is a simple solution: just re-generate the password until it satisfies all of the constraints. Although this might seem like a brute-force solution, in practice its speed is indistinguishable from a constant-time solution.

Below you have the new and improved YARPG:

I’ve also updated the original posting. You can get the source code for it by looking at the source of this webpage, or from my SVN repository: js_password_generator.html. Hopefully you find it useful!

Picture taken from cjc4454’s photostream with permission.

use strict;
use warnings;
use Net::SMTP::TLS;

my ($from, $password) = ('[email protected]', 'MySuperSecretPassword');
my $mailer = new Net::SMTP::TLS(
  'smtp.gmail.com',
  Hello => 'smtp.gmail.com',
  Port => 587,
  User => $from,
  Password => $password);

$mailer->mail($from);
$mailer->to('[email protected]');

my $data = <<'EOF';
X-Face: "8.]Z_3ptuNK'CA~DM>M,G.T(h=1.y9"0gXW3V91E:dw2?|&G2R(?/no'F2g4%8Fv.
 J1p5K-^1epKXxIG)mj4}nGWTi<=iz8n)bUVhLu}MXRFl9"J%'=-;IfMXcuPK>-%^;$uW87O/B
Subject: Hello X-Faced World!

email body.
EOF

$mailer->data();
$mailer->datasend($data);
$mailer->dataend();
$mailer->quit();

The code is largely based on this snippet: Sending Mail Through Gmail with Perl. The X-Face header was generated using the Online X-Face Converter (yes, I know that there is a Image::XFace module, but it was very cryptic – it didn’t mention supported input / output formats). One word of warning: if you are using ActivePerl under Windows, Net::SMTP::TLS isn’t available in the default module list (AFAIK, because of encryption restrictions), so you might need to experiment with alternative package sources or using Linux :-). I’ve also tested the script with an email account I control (using Thunderbird with the Mnenhy plugin – which can read but not create X-Face emails) and it worked nicely.

There you have it: how to use an old (from the 1980s according to Wikipedia) method for embedding pictures which is not supported by most of the email clients

If you are interested in the code samples (as you most probably are, since a big part of the presentation were demos), you can find them in my Google Code SVN repository. Feel free to contact me if you have questions about the slides or about something else in general.

Can you make the number 24 with the number 5, 5, 5, and 1 (again, you cannot join the numbers together, have to use each number once and only once, and are only allowed to add, subtract, multiply or divide them)?

And here is my brute-force solution:

permute(0, [5, 5, 5, 1], []);

sub permute {
  my ($partial, $numbers, $solution) = @_;
  
  if (0 == scalar(@$numbers)) {
    print @$solution, "n" if (24 == $partial);
  }
  else {
    for my $num (@$numbers) {
      my $mynums = [];
      my $skipped = 0;
      for my $mynum (@$numbers) {
        if ($num == $mynum && !$skipped) {
          $skipped = 1;
        }
        else {
          push @$mynums, $mynum;
        }
      }
      
      for my $op (qw(- + * /)) {
        my $mypart = eval "$partial $op $num";
        my $mysol  = [@$solution, $op, $num];
        permute($mypart, $mynums, $mysol);
      }
    }
  }  
}

The output is not very elegant and contains a decent amount of garbage (because it considers that we have a hidden zero at the start – ie. 0*5…) and also a lot of repetition (because it doesn’t take into account that 5 5 5 1 is the same as 5 5 5 1 with the first two numbers interchanged), but in the end it gives the correct answer:

... fake answers because it starts with 0 ...
*5+5*5-1
*5+5*5-1
/5+5*5-1
/5+5*5-1
*5+5*5-1
*5+5*5-1
/5+5*5-1
/5+5*5-1
*5+5*5-1
*5+5*5-1
/5+5*5-1
/5+5*5-1
... duplicate answers because of the order ...
-1/5+5*5
-1/5+5*5
-1/5+5*5
-1/5+5*5
-1/5+5*5
-1/5+5*5

Also the correct interpretation of the output is to consider that each operation has a pair of parentheses around them and not reading it according to the usual mathematical rules. Having this in mind the solution becomes:

((-1/5)+5)*5 = 4.8 * 5 = 24

Brute-force FTW