Grey Panthers Savannah

Category: python

  • Couple of thoughts about web crawling

    Couple of thoughts about web crawling

    These days crawling the web is becoming more and more difficult with websites trying to ensure that “only humans” access them. While there are some good arguments in favour of these efforts, I also have some good arguments: Anyway, here are some tips to help out in creating a crawler:

  • Proxying pypi / npm / etc for fun and profit!

    Package managers for source code (like pypi, npm, nuget, maven, gems, etc) are great! We should all use them. But what happens if the central repository goes down? Suddenly all your continious builds / deploys fail for no reason. Here is a way to prevent that: Configure Apache as a caching proxy fronting these services.…

  • Passing UTF-8 trough HTTP

    These days we should write every code as if it will be used by international people with a wide variety of personal information (just look at Falsehoods Programmers Believe About Names for some headscratchers). I would like to do add my small contribution to this by showing how UTF-8 encoded strings can be passed into…

  • Converting datetime to UTC in python

    So you need to convert a python datetime object which has a timezone set (“aware” in the Python nomenclature) to an UTC one with no timezone set (“naive”), for example because NDB on GAE can’t store anything else. The solution will look something like this: date = date.astimezone(tz.tzutc()).replace(tzinfo=None) For searcheability: the exception thrown by NDB…

  • GeekMeet talk about Google App Engine

    The GAE presentation I’ve given for the 12th edition of Cluj Geek Meet can be found here (created using reveal.js). You can find the source code here.

  • Running pep8 and pylint programatically

    Having tools like pep8 and pylint are great, especially given the huge amount of dynamism involved in Python – which results in many opportunities to shooting yourself in the foot. Sometimes however you want to invoke these tools in more specialized ways, for example only on the files which changed since the last commit. Here…

  • Clearing your Google App Engine datastore

    Warning! This is a method to erase the data from your Google App Engine datastore. There is no way to recover your data after you go trough with this! Only use this if you’re absolutely certain! If you have a GAE account used for experimentation, you might like to clean it up sometimes (erase the…

  • Using Jython from Maven

    This blogpost was originally posted to the Transylvania JUG blog. On the surface it looks simple: just add the dependency and you can run the example code. However what the jython artifact doesn’t get you are the standard python libraries like re. This means that as soon as you try to do something like the…

  • Detecting the Metasploit encryptors in one hour and 49 lines of Python

    I’ve seen a lot of blogpostings lately which proclaim that Metasploit payloads encrypted with one of the available encryptors and written into an executable file are somewhat “magically” capable of bypassing AV software (these posts usually contain a couple of VirusTotal links to demonstrate the point). The main scenario considered (from what I gather) is…

  • Negative zero – what is it?

    Computers have two ways of representing numbers: One is called sign and magnitude – usually you have one bit specifying the sign (again, most of time you have 0 for positive and 1 for negative) and the rest of the bits specify the absolute value (“magnitude”) of the number. The other is ordering the numbers…