mediawiki – Grey Panthers Savannah https://grey-panther.net Just another WordPress site Tue, 06 Sep 2011 09:36:00 +0000 en-US hourly 1 https://wordpress.org/?v=6.7.1 206299117 Quick’n’dirty Mediawiki file crawler https://grey-panther.net/2011/09/quickndirty-mediawiki-file-crawler.html https://grey-panther.net/2011/09/quickndirty-mediawiki-file-crawler.html#respond Tue, 06 Sep 2011 09:36:00 +0000 https://grey-panther.net/?p=56

URL='http://10.0.0.1' MIME='image/jpeg' 
  bash -c 'wget -q -O - "$URL/wiki/index.php?title=Special:MIMESearch&mime=$MIME&limit=500&offset=0" 
  | grep -Po "/wiki/images[^"]+" 
  | xargs -n1 -I {} wget "$URL{}"'

What it does: it uses the “MIME search” functionality on the wiki to locate files of a certain mime type and then xargs+wget each of them.

Limitations:

  • A maximum of 500 files are downloaded
  • Downloads are not parallelized, thus slower than they could be
]]>
https://grey-panther.net/2011/09/quickndirty-mediawiki-file-crawler.html/feed 0 56