gae – Grey Panthers Savannah

Cleaning up Google AppEngine Mapreduce Jobs

gpanther — Tue, 12 Nov 2013 10:31:00 +0000

Do you use the Google MapReduce library on AppEngine? And do you have a lot of completed tasks which clutter your dashboard? Use the JS below by pasting it into your developer console to clean them up! (use it at your own risk, no warranty is provided :-))


schedule = function() { window.setTimeout(function() { var c = $('a:contains(Cleanup)').first(); if (c.length > 0) { c.click(); } else { $('a:contains(Next page)').click(); schedule(); } }, 300); return true; }; window.confirm = schedule; schedule();

Converting datetime to UTC in python

gpanther — Thu, 07 Feb 2013 17:06:00 +0000

So you need to convert a python datetime object which has a timezone set (“aware” in the Python nomenclature) to an UTC one with no timezone set (“naive”), for example because NDB on GAE can’t store anything else. The solution will look something like this:

date = date.astimezone(tz.tzutc()).replace(tzinfo=None)

For searcheability: the exception thrown by NDB if you fail to do this is “NotImplementedError: DatetimeProperty updated_at can only support UTC. Please derive a new Property to support alternative timezones.”

Clearing your Google App Engine datastore

gpanther — Wed, 15 Aug 2012 07:08:00 +0000

Warning! This is a method to erase the data from your Google App Engine datastore. There is no way to recover your data after you go trough with this! Only use this if you’re absolutely certain!

If you have a GAE account used for experimentation, you might like to clean it up sometimes (erase the contents of the datastore and blobstore associated with the application). Doing this trough the admin interface can become very tedious, so here is an alternative method:

Start your Remote API shell

Use the following code to delete all datastore entities:


while True: keys=db.Query(keys_only=True).fetch(500); db.delete(keys); print "Deleted 500 entries, the last of which was %s" % keys[-1].to_path()

Use the following code to delete all blobstore entities:


from google.appengine.ext.blobstore import *
while True: list=BlobInfo.all().fetch(500); delete([b.key() for b in list]);  print "Deleted elements, the last of which was %s" % list[-1].filename

The above method is inspired by this stackoverflow answer, but has the advantage that it does the deletion in smaller steps, meaning that the risk of the entire transaction being aborted because of deadline exceeded or over quota errors is removed.

Final caveats:

This can be slow
This consumes your quota, so you might have to do it over several days or raise your quota
The code is written in a very non-pythonic way (multiple statements on one line) for the ease of copy-pasting