Grep a Tail to Monitor Your Access Log in Color

Here is a quick example of how to monitor your access log and colorize the IP address of the remote user:

tail -f /var/log/lighttpd/access.log | egrep --color=auto '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' 

It is crude, but works well enough (generally). The first part tails our webserver access log using the -f switch. This is piped to egrep with the –color=auto switch to colorize the portion we are want, which is an IP address. The result is that the IP address of the remote user of the request is colorized. This makes it easier while monitoring such logs in a terminal. It isn’t perfect and you can modify the regular expression to further foolproof it, but it works well enough for simple differentiation.

We can further adapt this slightly. Here are two more examples using this scheme. The first could filter out your local subnet, while the second filters out a local email client.

tail -f /var/log/lighttpd/access.log | grep -v "192.168.1." | egrep --color=auto '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' 

tail -f /var/log/lighttpd/access.log | grep -v "Thunderbird" | egrep --color=auto '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' 

As you can see, the we know pipe the tail to grep with the -v switch which will filter out any line entries that contain the quoted text. You could add anything here such as IP addresses, particular page requests, certain domains, etc.

Quick. Easy. Adaptable.

Using Python and Urllib2 to Access Pages with Basic Authentication

Need to grab material from a site that uses basic authentication? Here is a quick example of using Urllib2 and Python to do just that.

This is a quick bit of code that connects to protected pages on ‘Example.com’. We are going to go with the assumption that the page we want to grab data from, ‘orders_waiting.py’, simply contains an string representative of the number of orders we have awaiting in the queue. Therefore, we will grab the html, a digit as a string, replace any newline characters, and handle the integer value, reporting if it is a non-zero return. However, since the code important to this example is the authentication bit, I am only showing that portion here.

#!/usr/bin/python
import urllib2
SERVER = 'example.com/pass-protected/'
authinfo = urllib2.HTTPPasswordMgrWithDefaultRealm()
authinfo.add_password(None, SERVER, 'testuser', 'test-user-pass')
page = 'HTTP://'+SERVER+'/cgi-bin/tools/orders_waiting.py'
handler = urllib2.HTTPBasicAuthHandler(authinfo)
myopener = urllib2.build_opener(handler)
opened = urllib2.install_opener(myopener)
output = urllib2.urlopen(page)
do_something(output)

More on using Urllib2 can be found at the python documentation site here.

Cook your RAW photos into JPEG with Linux

NOTE: This is a ported post from the old Mutaku site. Original post information:
Thursday, August 14 2008 @ 04:50 PM EDT Contributed by: xiao_haozi

Shooting in RAW format is great for the photography buff. However, when it is time to share with others, post on your photo gallery, or print at the local photo printing shop, you want something more portable. Here we’ll look at converting RAW photos (specifically Canon’s CRW format) into JPEGs using a batch command-line approach in BASH.

We will be using two tools to do our batch conversion:

1- DCRAW (available here)

2- CJPEG (available here)

CJPEG is part of the standard JPEG library (libjpeg) so you should already have that installed. If not you can find the source here and built as such:

./configure --enabled-shared
 make 
 make install (as root, e.g. sudo make install)

DCRAW is available here and can be built as such:

wget http://www.cybercom.net/~dcoffin/dcraw/archive/dcraw-8.87.tar.gz
tar xzvf dcraw-8.87.tar.gz
cd dcraw-8.87
./configure
make
make install (as root, e.g. sudo make install)

Now that we have the prerequisites out of the way, we can try it out. We’ll manually do the conversion to ensure it works. Make sure you then compare the images and ensure they meet your needs/desires with the settings we use below.

In our example we will assume we have a directory called trip08 on our user’s desktop that contains some CRW raw images as well as others. We will go into this directory, make a CRW directory to copy our RAW images to for converting, search here and all subdirectories for CRW files and copy to our new directory, and then convert them using the two tools above so we will have in our new directory a copy of the original RAW CRW image as well as a new JPEG version.

Here we go:

cd ~/Desktop/trip08 && mkdir CRW_conversions
find . -name \*.[Cc][Rr][Ww] -exec cp "{}" CRW_conversions/ ";"

First, we went into our trip photo directory and made a directory called ‘CRW_conversions’ that we will work within. Next, we looked in the current trip directory (and subdirectories) for all files that end in ‘.crw’ including and variations on capitalization. The files we found are transfered to our CRW_conversions directory.

cd CRW_conversions
for i in *; do dcraw -c "$i" | cjpeg -quality 100 -optimize -progressive > $(echo $(basename "$i" ".crw").jpg); done

Now we went into our CRW_conversions directory that contains copies of our RAW images. We then constructed a loop which:

1- for i in *; do for each element in this directory do the following
2- dcraw -c “$i” | use dcraw to decode the element and pipe it out
3- cjpeg -quality 100 -optimize -progressive > accept the pipe from dcraw with cjpeg and convert to 100% quality jpeg with optimization and progressive processing and output that
4- $(echo $(basename “$i” “.crw”).jpg) take that output and put in a file called element.jpg where element is the name of the original CRW version without the extension ‘.crw’
5- ; done finish the loop once the iterations are done

We should now have a directory called CRW_conversions in our trip08 directory that contains copies of all our RAW images as well as fresh versions of each in JPEG format. The settings we used to convert have given us fairly high quality images that are still relatively large, so you may wish to alter these to your liking. You can do so by altering the CJPEG part of the conversion and find the options here:

man cjpeg

If you find that you have errors of ‘Empty input file’ and resultant JPEG files that are 0 bytes in size, you most likely have left out the ‘-c’ switch in your DCRAW part. This is needed to send the output with the pipe, or else you will end up with PPM files.

You can place most of this code into a shell script to do the brunt of the work for you without having to type in each time. We are currently working on a Python version which uses Tkinter to provide a GUI interface as well as options to convert and manipulate several image formats and will post source code and a how-to once it is finished. However, if you want to try out a shell version now you might want to try something like:

#!/bin/env bash
#######################################
# crw2jpg                                                           
#      convert canon raw into jpeg                       
#######################################
mkdir CRW_conversions
find . -name \*.[Cc][Rr][Ww] -exec cp "{}" CRW_conversions/ ";"
cd CRW_conversions
for i in *; do dcraw -c "$i" | cjpeg -quality 100 -optimize -progressive > $(echo $(basename "$i" ".crw").jpg); done
print "CONVERSIONS COMPLETE"

This is a very basic script without any real error control or logic, so add to it as you wish/see fit. If you wish to keep as such, you should just make sure that you don’t already have a conversion that was run in the existing directory. To run this:

#(copy the above script)
nano ~/bin/crw2jpg
#(paste and save)
chmod +x ~/bin/crw2jpg
cd to_directory_with_new_photos/
crw2jpg

Enjoy cooking those RAW images, and be sure to check back for our conversion GUI. Good luck!

Backing up a personal DVD

NOTE: This is a ported post from the old Mutaku site. Original post information:
Sunday, January 02 2011 @ 12:18 AM EST Contributed by: xiao_haozi

Sometimes you find the need to backup a personal DVD. There are many GUI tools available that are up to the task, but sometimes some good, old command-line magic is just what you need to get the job done… especially with some more troublesome tasks.

I recently received a personal video that had been burned to a dual layer DVD and region coded. I wanted to make a copy for myself but had some trouble getting the DVD to open in tools like K3B and K9Copy. So I went to some old command-line magic to get things done the simple and old school way.

I will simply outline each step quickly to give you an idea of how things are done and let you tweak things to meet your situation. Hopefully, things will be pretty evident where changes will need to be made to meet your personal setup such as device identifiers, file names, and audio/video quality preferences.

Also note, this is for personal, non-encrypted DVDs. This won’t work on commercial DVDs that use CSS based encryption and isn’t a tutorial intended for those purposes. This tutorial is aimed to backup your personal videos.

To briefly summarize, we will first do a direct copy of the DVD drive contents and make a temporary file with our project name “this_project”. We then will encode the contents using dvd quality parameters with mencoder (and these can be adjusted to suit your own space/quality limitations and desires). After creating the new DVD project with dvdauthor, giving use the appropriate video_ts and audio_ts directory structures, we write this to a DVD with growisofs.

The actual rip part:

dd if=/dev/sr0 of=this_project.iso

The mencoder part (this is all a one-liner) to create the DVD quality mpeg:

mencoder -oac lavc -ovc lavc -of mpeg -mpegopts format=dvd:tsaf\
 -vf scale=720:480,harddup -srate 48000 -af lavcresample=48000  \
 -lavcopts vcodec=mpeg2video:vrc_buf_size=1835:vrc_maxrate=9800:\
vbitrate=5000:keyint=18:vstrict=0:acodec=ac3:abitrate=192:autoaspect -ofps 30000/1001\
 -o this_project_dvdqual.mpg this_project.iso

Making the iso- create this xml called this_project.xml… here we won’t make any chapters:

<dvdauthor>
  <vmgm>
    <titleset>
      <titles>
        <pgc>
          <vob file="this_project_dvdqual.mpg">
        </pgc>
      </titles>
    </titleset>
</dvdauthor>

Now we will author the DVD with the structure we called for in the xml file and then burn the DVD iso image:

dvdauthor -o this_project -x this_project.xml
growisofs -dvd-compat -Z /dev/sr0 -dvd-video ./this_project_dvd/

As you can see, it’s a pretty simple process. The most intimidating step is creating the temporary mpeg file with mencoder, but you can easily follow the writeup here if you wish to change any of the parameters. However, the ones I have chosen and outlined above should give you a pretty nice standard NTSC DVD format video. It’s really not that bad.

Copy and Paste to and from the Shell with the Cr-48

Copying and pasting to and from the Cr-48 shell can be a bit tricky when using only the built-in hardware (i.e. no usb mouse). Read on for a quick example of copying text from the browser to a shell session and vice versa.

Copying text from the browser to the shell:

  • Highlight your text of interest in the browser (to highlight by dragging on the touchpad, simply double tap the touchpad with one finger and then move to highlight the text).
  • Copy the text by right clicking (e.g. two finger clicking on the selected text).
  • Switch to your shell (ctrl-alt-t).
  • Paste the text by using three fingers to click the touchpad (it can be a bit tricky on the recognition end).

Copying text from the shell to the browser:

  • Highlight your text as outlined above.
  • Now to copy, don’t do anything further in the shell, simply get back to the browser (ctrl-alt-t).
  • In the browser now, paste anywhere you wish by initiating a three finger click on the touchpad (or doing a quick two finger click twice).

Setting up a linux dialup connection

NOTE: This is a ported post from the old Mutaku site. Original post information:
Tuesday, June 19 2007 @ 06:20 PM EDT Contributed by: xiao_haozi

My dirty walk-through to setting up an external modem in Linux using wvdial to dial into an ISP connection.I had some trouble finding a single place with a good write-up for doing such a task when I was setting up a Linux box for someone that had previously been using Windows and a little OS X. Finding dialing info, modem configuration, or slight troubleshooting, alone wasn’t too difficult, but finding them in one place in a concise manner was. So I have attempted to compile what I have found, my tweaks and adjustments, and other hints that enabled me to setup a Linux box to use a dial up connection via external modem.

As I mentioned, information for this task was extremely piecemeal, and finding any form of updated support for a modern Ubuntu distribution (Feisty Fawn – Kubuntu in particular) was like searching for a pay phone in Hollywood, or so I imagine. Similes aside, most of the information I could locate was for workarounds with internal winmodems (or lack thereof), or even the, “…yeah, why are you still trying to use dial up?” Regardless of the archaism of the technology, America is apparently still clinging to this method of exploration as evident by googling some statistics on broadband penetrance.

So there I was left with a monumental task for my new school networking knowledge (setting up firewalls, DHCP, NAT, whipping up some cat5, etc) and burgeoning Linux skills. In the end, I could hear that clunky, mystical modem speaker cranking away, and yes, the sweet, sweet splendor of waiting eons for those jpegs to load in my Google image search. Away we go —

The hardware and tools I had available were as follows (with a little explanation why where relevant):

* Linux box (AMD64) : trying to switch them to a free framework (and that which I find most fun)

* Kubuntu Feisty Fawn 7.04 (64bit version) : running KDE of course (I find KDE to be a little more user friendly for those coming from windows — bring on the flame war — but that is just mho)

* A few sporadic guides (I truly apologize for my lack of references but it I will try to add a list of sorts at the end)

* external modem : Trendnet TFM-560x v.92 56k serial modem (because internals are a pain to find — at least in my house — and most of mine were winmodems from scrapped dell boxes which use proprietary drivers from conexant — and because external modems are sexy)

* PCI serial card : Syba PCI multi I/O PCI 2 serial nm9835cv netmos dual ports (because the Asus motherboard I was building with didn’t have a serial port onboard — only Parallel — and I didn’t want to fight with a serial2usb approach)

* some command lines tools you will see during the guide

[WARNING: the guide to follow is what worked for me and is probably pretty crude and round-about. If you have a much more coherent and pretty way to do this, please leave comments for the others and myself. I am merely a computer hobbyist and a professional in any sort of computer related field by no stretch of the imagination (biochemistry to be specific) and accordingly this may be a little crude and ugly to some. Like I said, my Linux skills are not Jedi so bear with me please!]

First we need to find out where the PCI card is located and what the address of the serial ports are — we can do this many ways but I like to first check to see that the card is recognized and the like:

$ lspci -vv
 

We are looking for a ‘netmos’ PCI card and should see some info about it there with 5 regions listed all under the serial controller. You can also try scanpci if you wish.

Now we need the hex port address for the 2 serial ports on our card — I like to use dmesg and scan for it:

$ dmesg | grep "irq = 17"
 

You should see ttyS2 and ttyS3 entries — for example:

0000:04:09.0: ttyS2 at I/O 0x9c00 (irq = 17) is a 16550A
 

‘ttyS2’ tells us where — ‘0x9c00’ tells us the hex-port ID — and ‘16550A’ tells us the type.

 

So we can rerun this command by hitting up and then appending an output to save it to a file for reference:

$ dmesg | grep "irq = 17" > port_locations
 

We next need to install the tool to set the serial ports if we don’t already have it:

$ sudo apt-get install setserial
 

Since we now know where our PCI and serial ports are located we can set them to be used as follows by referencing our port_locations file we outputted the grep to (note: I have assigned both ports to the modem for ease in case it gets switched when I give it to the users. However, if you are going to be using this for something else keep that in mind — I haven’t tested it out, but I would make the broad assumption that setting both serials now shouldn’t affect adding in a scanner, let’s say, as long as the speeds are compatible and we have told the /dev/modem link to look at the right one):

$ setserial /dev/ttyS2 port 0x9c00 UART 16550A irq 17 Baud_base 115200

$ setserial /dev/ttyS3 port 0x9800 UART 16550A irq 17 Baud_base 115200
 

Now we will check that each is linked appropriately:

$ setserial /dev/ttyS2 -a

/dev/ttyS2, Line2, UART: 16550A, Port: 0x9c00, IRQ:17

Baud_base: 115200, close_delay: 50, divisor: 0

closing_wait: 3000

Flags: spd_normal skip_test

$ setserial /dev/ttyS3 -a

/dev/ttyS3, Line2, UART: 16550A, Port: 0x9800, IRQ:17

Baud_base: 115200, close_delay: 50, divisor: 0

closing_wait: 3000

Flags: spd_normal skip_test
 

Since we have plugged in our modem into the left most (closest to the motherboard) we are using ttyS2 so we will link this to /dev/modem for our dialer program to use (of course this is if you plugged your modem into the serial port closest to the mobo):

$ sudo ln -sf /dev/ttyS2 /dev/modem
 

and check it:

$ ls -l /dev/modem

lrwxrwxrwx 1 root root 5 2006-05-11 15:40 /dev/modem -> /dev/ttyS2
 

On reboot, this link we just created will be reset to the original location, i.e. something other than where our modem truly is located. So we need have this linked on startup and we can therefore create an init.d script to do such:

$ cd /etc/init.d/

$ sudo nano modem-link

#!/bin/bash

# modem-link : fix the /dev/modem link on startup

ln -sf /dev/ttyS2 /dev/modem
 

Set the execute bit and allow it to run at startup:

***** IMPORTANT: don’t forget that period after 99 2 3 4 5 when doing the update-rc.d below *******

$ sudo chmod +x modem-link

$ sudo update-rc.d -f modem-link startup 99 2 3 4 5 .
 

Now ones inclination would be to fire up KPPP or GPPP and use the gui to configure the dial up options. However, (at least in my build — and apparently most — see ubuntuforums.org) KPPP does not work. I managed to get it so far as to see the modem and attempt to dial, however the dial sequence was weird and would not dial properly. I really wanted a gui for this part because the users I was building this for would be dialing up and are not comfortable with the command line yet. And I wanted things to be simple – I didn’t want connecting to the internet to scare them from Linux back to windows or OS X. However, I knew that the easiest way (i.e. the most trouble-free method) would be to use a command line dialer and then script it somehow, so for now there interaction with the terminal would be a little less daunting. Let’s move on to my favorite command line dialer, wvdial.

Let’s get wvdial if we don’t already have it:

$ sudo apt-get install wvdial
 

Now we need to give wvdial our ISP information in the config file:

$ sudo nano /etc/wvdial.conf

[Dialer Defaults]

Phone = (your ISP number)

Username = (your ISP username)

Password = (your ISP password)

New PPPD = yes
 

Now lets use a KDE style launcher to do the launching of wvdial:

Open the Kmenu for editing and choose to add a new program.

We will name it (isp_name_here)-dialup, choose an icon of your liking, and then under the execution portion enter:

sudo wvdial
 

also choose to run in a terminal.

Now save it and exit.

To dial up now, all we have to do is find our little launcher we just made in the Kmenu, run it, and it will prompt us for our user password. Type in the password for the current user and you will see wvdial running in a terminal window. Once you see that you have acquired your IP address it will pause there and you should be ready to roll. Minimize your terminal window that wvdial is in and off to the races you go (well maybe not races as we are using dial up — maybe off to bingo?! we go).

Now to exit from our internet excursions and get our phone line back — maximize the terminal window and type ctrl-c (control + c) and this will exit wvdial and thus the connection.

So there you have it: a possibly very round-about way to get your dial up going. I found it to be a very elegant method. However, I am sure that reading through this, many have cringed, spit coffee at their monitors, or deleted my page from their bookmarks. But it worked for me and taught me some fun things about port assignment, writing startup scripts, and the like. I would appreciate your feedback: what worked, what didn’t work, did you do it another way, etc.

Good luck — modems ahoy~!

Forcing Your Monitor to Take a Nap

I usually don’t like power management touching my display devices. Things just never seem to function to meet my changing (by the minute) needs. I avoid screen savers as they usually drain far too many resources and come on and off at the least opportune moments.

This is a nice take it to the man kind of approach to sleeping your monitor on-demand. No screen saver, but all the benefits of blanking the screen. Just create a file with the following code and then you can go into your window managers key assignment menu and assign a keyboard press to run the script. I generally map all my machines’ pause/break key to do so.

#!/bin/bash

# set the monitor to sleep using xset
xset dpms force standby

-Matthew

Tempel 1 NASA Pass-by

Tempel 1 NASA Pass-by

 

Synopsis:

This is a brief project involving NASA’s STARDUST-NExT spacecraft. In 2006, the craft collected samples from Comet Wild 2 and turned back around on a follow-up mission to visit the comet Tempel 1. On February 15 of this year (2011), a series of images of a Tempel 1 fly-by were posted on the Jet Propulsion Laboratory website. Blown away by such an amazing event captured in stunning clarity, I immediately had to put together an animated version of these images. Below is a short summary of a quick hack to create such an animation as well as the resulting files.

More Information:

For more information on the mission, feel free to peruse the press-kit information or visit the fly-by page.

The hack:

Python Urllib2 and Urllib are used to grab the pages and download the images, respectively [urllib2,urllib]. Images are then processed with ImageMagick. NOTE: Using a parsing library like Beautiful Soup would be much better, but what would a hack be if it were the optimal method? Now to the code…

grabber.py – get the images download

This will dump all the images into the current directory so probably best to make a new directory and launch from within that desired location.

#!/usr/bin/env python

import urllib2,urllib

lines = urllib2.urlopen('http://www.jpl.nasa.gov/news/stardust/').readlines()

files = []
for line in lines:
	if '<a href="/news/stardust/index.php?fileID=' in line:
		files.append(line)

links = []
for line in files:
	if "<h3>Comet Tempel" in line:
		links.append(line)

pre = "http://www.jpl.nasa.gov"
linkslist = []
for line in links:
	this_l = line.split('"')[1]
	linkslist.append(pre+this_l)

x = 0
for link in linkslist:
	thislink = link
	thispage = urllib2.urlopen(thislink).readlines()
	images = []
	for line in thispage:
		c = []
		if "../../images/stardust/stardust2/" in line and "alt='Comet Tempel 1" in line:
			c.append(line)
		if c:
			images.append(c[0])
	for image in images:
		d = image.split("'")[1]
		g = thislink.rpartition("index.php")[0]+d
		thisimage = g.rpartition('/')[2]
		x += 1
		try:
			image = urllib.URLopener()
			image.retrieve(g,thisimage)
			print "Grabbed image # ",x," ==> ",thisimage
		except:
			print "**** Problem grabbing image # ",x," ==> ",thisimage

Making the animations:

The following will invoke a small ImageMagick GUI instance. Once the images are loaded the animation will play. To tweak/save, right click and the menu will appear where you can manipulate the GIF. Seen here.

animate *.jpg

Making a scaled version:

Use ImageMagick to scale all the images and animate the scaled products.

for i in *.jpg; do convert -scale 20% "$i" $(basename "$i" ".jpg")"_scaled.jpg"; done
animate *_scaled.jpg

Results:

Okay, enough for you already? Well here are the results: Large version and Scaled version.

 

… Then felt I like some watcher of the skies
When a new planet swims into his ken;
Or like stout Cortez when with wond’ring eyes
He stared at the Pacific …

– John Keats, ms of sonnet (1816)

-Matthew