Category Archives: Hacks

Quick and (usually) simple hacks.

Accessing Django Template Variables in Javascript

How do we follow the Django DRY principle and avoid as many hard coded elements as possible in our project when we need to include Javascript? A good example of this is if we are pulling in a bookmarklet that will be offered on the site. We need to reference the domain of our project in the bookmarklet, but this then yields code buried in our project’s static files that needs to be altered by anyone that might be utilizing your open sourced code.

This example presents a little bit of a challenge. If this were HTML in our template we could simply reference a DOMAIN variable that we would feed in through a view. However, this is a static Javascript file, so how do we go about utilizing Django template variables in supposedly static Javascript?

Here is a simple method. In our HTML template, we will setup a Javascript global variable that is created to reference a template variable that we feed in from our view, here a domain, and this variable is then referenced in our static Javascript files.

<!-- Our HTML Template (templates/index.html) that is fed data from views.py -->
{% extends "base.html" %}
{% block content %}

<script type="text/javascript" src="/static/somejs.js"></script>

<script> 
var DOMAIN = "{{ site_domain }}";
</script>

<!-- rest of our template, business as usual. -->
{% endblock %}

As you can see, we would feed in a site_domain variable from views.py that might be created something like this.

from django.contrib.sites.models import Site
from django.shortcuts import render_to_response

def index(request):
    '''Our main index view.'''
    site_domain = smart_str(Site.objects.get_current().domain)
    # more of our view code...
    
    return render_to_response('templates/index.html', {
	'site_domain': site_domain,
        #...rest of our template vars
    })

Now, how do we use this in our static Javascript? We reference that global Javascript variable we declared in our template using our Django template variable site_domain.

// Our Javascript file ... /static/somejs.js 
alert("Your current domain is: "+DOMAIN);

Now our code is much more portable, even in places where things should essentially be static.

Setting Default Browser in Linux When Things are Stubborn

Running XFCE4 and setting the preferred applications generally works. However, I noticed that some applications still would not listen to my demands of using a specific browser when launching links. For example, using python’s webbrowser module, and therefore almost any other module opening external links, would use Firefox or Opera (depending on which was available).

# Checking python invocation of a browser
import webbrowser as w
w.get().open('http://mutaku.com')

Here’s the culprit, check in your /etc/alternatives/ directory and you’ll see what I mean.

ls -l /etc/alternatives/

Notice the entry x-www-browser. This will be a symlink to a browser and this value was different than what I was attempting to set through the XFCE4 GUI configuration. This problem seems somewhat common as I’ve seen it occur in KDE and GNOME as well.

To fix this issue, simply run the following command and you should be presented with a list of browsers from which you can set.

# Update your browser choice and choose from the presented text menu
sudo update-alternatives --verbose --config x-www-browser

# Now make sure teh proper symlink is setup
ls -l /etc/alternatives/x-www-browser

Now you should have a functional default choice. Those GUIs can suck it!

Transposing Matrices in Python

Situation:
We have a list of nested lists (list of matrices) that we wish to transpose.

Solution:
Utilize chain to iterate over our master list (x), flattening it, and then we iterate over the elements and return these as a transposed list element.
We can also check that an iterated element exists and meets some value requirement (here, less than 200).

from itertools import chain
[[y[i] for y in list(chain.from_iterable(x)) if y[i] and y[i] < 200] for i in range(0,len(x[0][0]))]

Caveats:
Here, we are assuming that the subunits are invariable in length.

Analyzing Brute Force Attempts in BASH

If you have a public facing SSH server running on the standard port, your message log is probably filled with failed authentication attempts from brute force attacks. Let’s mash up some quick BASH commands to analyze this data. For our purposes, we’ll look at the top attacker IPs and the top usernames tried.

First, pull down all of your message files and decompress them in a working directory using bunzip2.

Once you have all your message logs ready, we will search through them and pull out all of the authentication failure entries and grab the IP and username for each attempt.

grep -r "authentication error" messages* | awk '{split($0,a," "); print a[NF],a[NF-2]}' > attempts

So we first use grep to look for failed authentications by searching recursively for the string “authentication error” in all of our message logs by using the wildcard. We then pipe this to awk and split each input line found into an array delimited by a whitespace. The last part of each line, and therefore our new array, goes something like: ‘authentication error for USERNAME from IP’. So to get the username and IP from our array we can use the array length variable NF and use that to index the variables we need. Here we grab the last using a[NF] and two in from that with a[NF-2]. Finally, we output this to a file called attempts.

Now, let’s use more BASH magic to do the analyses for us. Our attempts file now is in the format as follows: IP USERNAME. We want to see the top IPs and usernames and we can do that with some sorting commands.

cut -d ' ' -f2 attempts | sort | uniq -c | sort -nr > attempts_username
cut -d ' ' -f1 attempts | sort | uniq -c | sort -nr > attempts_ip

Here, we simply grab either the username in the second column or the IP in the first with cut, sort this data, prepend lines with the number of occurrences of each, and then sort by this occurrence number and output to a new file. You can now view attempts_username and attempts_ip to see the top usernames and IPs, respectively, of brute force attacks.

Lastly, we can associate the keep usernames and IPs together and sort on one or the other to see the correlation between the two. To end our initial analyses, we will sort on usernames and find out for the top attempted usernames, what are the top originating IPs.

 sort -k 2,2 attempts| uniq -c | sort -nr | head -n 10

Next time we will be using some GEOIP methods to see where our top attack attempts are originating.

Finding Images with Python

Let’s say we wish to take a directory, the current directory in this example, and gather all the image files. This could be helpful if we were then going to use PIL or Image Magick to manipulate those files, or create a select-able list for use in a GUI, and so on…

There are countless ways we could do this, but here is one with a module that is very helpful with image details.

import os,imghdr
for f in os.listdir(os.getcwd()):
	try:
		if imghdr.what(f):
			print f
	except:
		pass

Bash Snippet for Git Users

Keeping track of what branch you are using while in a git repository can be a bit of a pain. Adding the following code to your ~/.bashrc file can make it a bit easier since it will prepend your prompt with the current branch.

function parse_git_branch {
    GIT_BRANCH=$(git branch --no-color 2> /dev/null | awk '{if ($1 == "*") { printf "(%s) ",$2}}')
    PS1="$GIT_BRANCH\u@\h:\w\$ "
}

PROMPT_COMMAND="parse_git_branch"
Here it is working:
matt@desktop:~$ source /etc/bash.bashrc 
matt@desktop:~$ cd programming/bits
(master) matt@desktop:~/programming/bits$ 

Python – Dynamically Printing to Single Line of Stdout

It’s that time again; code snippet time. This is a short and sweet bit of a code that has some really practical and powerful implications.

When handling large amounts of data processing, it’s often desirable to keep the user informed of where things are in terms of progress. However, filling up the scrollback buffer with thousands of “Completed processing [item X]” or “X % completed” is not always desirable. So, how do we handle verbosity without filling scrollback?

Here, we will print data using stdout but we will continually use the same line of the terminal. It’s extremely simple; before each data print, we will move the cursor back to the beginning of the current line, cut out the entire line of text, and then print our current bit of data.

*Note: This is using ANSI escape sequences and therefore will work on any VT100 type terminal.

 

The code:

import sys

class Printer():
	"""
	Print things to stdout on one line dynamically
	"""

	def __init__(self,data):

		sys.stdout.write("\r\x1b[K"+data.__str__())
		sys.stdout.flush()

 

To output data we might do something like:

totalFiles = len(fileList)
currentFileNum = 1.0

for f in fileList:
	ProcessFile(f)
	currentPercent = currentFileNum/totalFiles*100
	output = "%f of %d completed." % (currentPercent,totalFiles)
	Printer(output)
	currentFileNum += 1

 

And here is an example of what our output would look like in a practical scenario.

Processing some data

Progressing along with dynamic status updates

Simple. You can test it out using an 'for x in range(1,100)' sort of statement as well. Now we can keep a persistent display of data processing or program progress without slamming the scrollback buffer of the user's terminal.

Installing Google Chrome in Slackware 13 (13.37)

Trying to install Google-Chrome via RPM (with rpm2tgz) will give you an error with libnss3. Don’t fret, there is already a way to build Google-Chrome without much effort. Below is a quick way using the mirror from Penn State (carroll), but if you have your install media, you can find the same files there in /extra.

Go to: ftp://carroll.cac.psu.edu/pub/linux/distributions/slackware/slackware-13.37/extra/google-chrome/

The README file is pretty self explanatory, but essentially you want to do the following:

  1. Become rootSetup a build directory (mkdir chrome && cd chrome)
  2. Wget these files: [ ftp://carroll.cac.psu.edu/pub/linux/distributions/slackware/slackware-13.37/extra/google-chrome/google-chrome.SlackBuild ] [ ftp://carroll.cac.psu.edu/pub/linux/distributions/slackware/slackware-13.37/extra/google-chrome/google-chrome-pam-solibs-1.1.3-i486-1.txz ] [ ftp://carroll.cac.psu.edu/pub/linux/distributions/slackware/slackware-13.37/extra/google-chrome/ORBit2-2.14.19-i486-1.txz ] [ ftp://carroll.cac.psu.edu/pub/linux/distributions/slackware/slackware-13.37/extra/google-chrome/GConf-2.32.1-i486-1.txz ]
  3. Change execution bit on slackbuild file and run it (chmod +x google-chrome.SlackBuild && ./google-chrome.SlackBuild)
  4. Go to tmp and install the file (cd /tmp && upgradepkg –install-new google-chrome.txz)
  5. Go back and install the other files (cd /root/chrome/ && upgradepkg –install-new *.txz)

Now you should have a fully functional google-chrome install.

Better Trackpad Functionality with CR-48 in Ubuntu

There is a great post on patching up synaptics based on what is found in opensuse here.

Takes all of 5 minutes and after patching and building the packages, a quick install, and then logging out and back in, you can now use a more functional cr-48 trackpad with your dev-mode ubuntu install.

Confirmed working in 11.04.

Uploading Images to imgur with Python

That time again. Code snippet time.

UPDATE: See the bottom of the post for the most up-to-date version of the code.

Another brief one. We are going to start working on a class to upload images to imgur.com. There are many ways we can go about doing this, but to start things off, we’ll use the simplest way which is to use the public/anonymous API. This means that anyone can view our images and that we do not need to worry about authentication just yet.

We will use pycurl for the url handling and parameter passing and to decipher our results we will utilize the XML minidom.

Let’s jump right in and see the code. You will need an API key, available here, that you will need to substitute for the key parameter string “YOUR_API_KEY_HERE”.

class UploadImage():
	"""
	Upload images to imgur
	"""

	# Not very verbose at the moment in terms of errors - will build on that later
	def __init__(self,image,upload_method="anon"):
		import pycurl
		from xml.dom import minidom
		import cStringIO

		# setup some initial items we will need
		self.c = pycurl.Curl()
		self.response = cStringIO.StringIO()
		self.minidom = minidom
		self.image = image
		self.imageURL = ""
		self.error = ""

		# provide some various methods for uploading
		# by default - at least for now in testing - we will use anonymous upload method
		if upload_method == "anon":
			self.anon_upload()
		# later or on-demand we can switch to oauth based uploads
		elif upload_method == "oauth":
			self.auth_upload()

	# anonymous upload method
	def anon_upload(self):
		"Upload anonymously to imgur"

		# setup the basic parameters
		params = [
				("key", "YOUR_API_KEY_HERE"),
				("image", (self.c.FORM_FILE, self.image))
			]

		# setup the url and pipe in our key and image
		self.c.setopt(self.c.URL, "http://api.imgur.com/2/upload.xml")
		self.c.setopt(self.c.HTTPPOST, params)

		# we want to capture the output so lets set the write output to go to our cStringIO so we can parse it
		self.c.setopt(self.c.WRITEFUNCTION, self.response.write)

		# run it
		self.c.perform()
		self.c.close()

		try:
			# parse the XML return string and get the URL of our image
			xml = self.minidom.parseString(self.response.getvalue())
			self.imageURL = xml.getElementsByTagName("original")[0].firstChild.data

		except:
			self.error = "Problem uploading anonymously."

		return self.imageURL,self.error

	# oauth-based upload method
	def oauth_upload(self):
		"Upload using oauth to imgur"

		### Not coded yet but we will use python-oauth2
		pass

To use our uploader, all we have to do is something like the following:

In [1]: from handlers import UploadImage as U

In [2]: a = U('roosevelt.jpg')

In [3]: a.imageURL
Out[3]: u'http://i.imgur.com/cUWzn.jpg'

In [4]: a.error
Out[4]: ''

As you can surmise, we could simply check that there was no error and then utilize the returned URL of our newly uploaded image. Pretty simple stuff and very handy for some quick public/disposable image uploads.

Next time, we will expand our uploader to include an oAuth method that we can utilize for private uploads. Additionally, we will explore our XML returns a little deeper for methods to delete our files or grab some basic view statistics.

Updated code: Here, we are using things in the context of Django, hence the settings call for pulling in the API key and so forth.

class UploadImage():
	"""
	Upload images to imgur
	  returns either .error or .imageURL

	TO DO:

	POSSIBLE TO DO:
		- add stats and image info functions
		- allow for full JSON responses instead of having to manually parse XML

	"""

	# Not very verbose at the moment in terms of errors - will build on that later
	def __init__(self,image="",dhash="",delete=False):

		# setup some initial items and placeholders we will need
		self.c = pycurl.Curl()
		self.response = cStringIO.StringIO()
		self.minidom = minidom
		self.image = image
		self.dhash = dhash.__str__()
		self.delete = delete
		self.message = ""
		self.imageURL = {}
		self.error = []

		if self.dhash and self.delete:
			# if we have a hash and a delete trigger, lets try to wipe the image
			self.wipe()

		else:
			# fire away an upload - we will return an imageURL dictionary with attributes or an error
			self.upload()


	def upload(self):
		"Upload anonymously to imgur"

		# setup the basic parameters
		params = [
				("key", settings.imgur["anon_key"]),
				("image", (self.c.FORM_FILE, self.image))
			]

		# setup the url and pipe in our key and image
		self.c.setopt(self.c.URL, "http://api.imgur.com/2/upload.xml")
		self.c.setopt(self.c.HTTPPOST, params)

		# we want to capture the output so lets set the write output to go to our cStringIO so we can parse it
		self.c.setopt(self.c.WRITEFUNCTION, self.response.write)

		try:
			# run it
			self.c.perform()
			self.c.close()

		except:
			self.error.append("Problem uploading image.")

		if not self.error:
			# parse the xml
			self.parseXML()


		return self.message,self.imageURL,self.error


	def wipe(self):
		"Wipe an anonymouse image from imgur"

		deleteURL = "http://api.imgur.com/2/delete/%s" % self.dhash

		self.c.setopt(self.c.URL, deleteURL)

		self.c.setopt(self.c.WRITEFUNCTION, self.response.write)

		try:
			self.c.perform()
			self.c.close()

		except:
			self.error.append("Problem deleting image.")

		if not self.error:
			self.parseXML(delete=True)


	def parseXML(self,delete=False):
		"Parse the XML ouput from IMGUR and write to the imageURL dictionary"

		try:
			# parse the XML string into the dom
			xml = self.minidom.parseString(self.response.getvalue())

			if delete:
				self.message = xml.getElementsByTagName("message")[0].firstChild.data

			else:
				# grab out some helpful/interesting data and setup in the imageURL dictionary
				self.imageURL['url'] = xml.getElementsByTagName("original")[0].firstChild.data
				self.imageURL['hash'] = xml.getElementsByTagName("hash")[0].firstChild.data
				self.imageURL['deletehash'] = xml.getElementsByTagName("deletehash")[0].firstChild.data
				self.imageURL['smallthumb'] = xml.getElementsByTagName("small_square")[0].firstChild.data
				self.imageURL['bigthumb'] = xml.getElementsByTagName("large_thumbnail")[0].firstChild.data

		except:
			self.error.append("Problem parsing XML output.")