Apr 282016

Emrer is a Python script that reads a YAML file and starts an EMR cluster as specified in that file.

The main advantage over other EMR automation solutions is that it will take care of uploading the bootstrap/step scripts to S3, so everything can be stored locally, both the cluster config and the scripts it’s going to execute. Which basically means that a cluster created with this script can be stored in a versioning system like Git and basically treated as code all the way.

The configuration file is YAML, easier to read and understand than JSON. The example configuration is commented.

It’s not using CloudFormation at all, when this script was initially written CloudFormation didn’t yet know how to create EMR clusters. At the time I didn’t find anything else that could do it out there either.

It could be enhanced with a kind of “plugin” system where custom procedures are executed when certain things are set up. For example, a procedure that would add required security groups to the list if they are missing from the configuration file, making sure that the cluster is compliant with company regulations.

Feb 242014

My ideea was to set up a script running on Google’s servers that would automatically get files from my site and back them up on Google Drive. I would have gotten a free off-site backup. Unsurprisingly, it’s not working due to limits imposed by Google, even though I did manage to cheat on the 10MB limit for UrlFetchApp and file creation. Instead it now fails either because it’s execution takes too long, or because it tries to write to Drive too many times in a short period.

Some basics first.

What is Google Apps Script? JavaScript based, it can be used to make Google’s cloud do stuff for you. Basically web apps can be written that run on Google’s servers and use Google products like Maps or Calendar.

What is Google Drive? (Really?) It’s a site where people can store files. 15GB for free. Like Dropbox.

There are limits though and they are set pretty low. Understandable, as it would be easy to abuse otherwise. Like, for example, the maximum size that can be fetched from an URL is 10MB. The maximum size of a file that is created via scripting is also 10MB. Not really useful for backups.

Right. On to the actual script and why it (still) doesn’t work.
Continue reading »

Nov 252011

According to the comments this is supposed to be a very basic echo server. It was written for school many years ago and I didn’t turn out to be a C coder after all, though I remember this being fun, so for anyone stumbling upon this: DISCLAIMER – apparently it was six in the morning and looks like I was a bit drunk too, so, seriously, don’t try it if you don’t know what it does. You shouldn’t run unknown code on your system, specially C code found on some random site, if anything happens, I don’t care. Just in case this does screw something up, drop me an email so I can laugh at you. Copy&pasted verbatim for archival purposes. Other than that it’s perfectly safe and no one lies on the internet.
Continue reading »