Transparent legend in your plots

Have you ever made a plot with several data series together and found that there is no way to put an explanatory legend onto the plot without hiding crucial data points? One answer to this problem can be to overlay a transparent legend. There are two routes to this "graphical goodness" that I would immediately think about:

• Create an image file from your plot and then "photoshop" it (or gimp it...)
• Add the alpha channel directly in the plotting tool...

Matlab can possibly do this, I really don't know. Here's how to do it in python. Say you have some data you want to plot, for example the sine and cosine function. Then you'd overlay a transparent key to your data. Let's dive in..

from pylab import * t = linspace(-2*pi,2*pi) y1=cos(t); y2=sin(t); plot(t,y1,label='Cosine') plot(t,y2,label='Sine') xlabel('Time') ylabel('y')
axis([-2*pi,2*pi,-1,1])
leg=legend()
savefig('namehere.png')
close()

Now we see from the resulting plot that the legend covers some of the graph. Now, before the "savefig" command, add the following:
frame=leg.get_frame()
frame.set_alpha(0.4)

The resulting plot is shown below the original. Better, don't you think? :-)

PS! Click on the plots to show larger versions!

Coding is better with java ... the coffee, that is.

Ah... a non-tech, short post. This morning I started implementing some numerical experiments. Sitting in my favorite coffee shop drinking the best brew I know - that I call perfect conditions for coding. And, as always, Python is your friend (Java is a coffee type, not a programming language... ).

So what did I do in the coffee shop? I wrote a small reporting tool to compute total hours spent on different projects as registered in the timereg tool I described previously. The tool is very simple; I just read the timereg log into a Python list using the readlines command and exploit the structure. I make two lists; one for project names and one for hours spent. Each time a project name is encountered that I haven't seen before when iterating over the readlines list, I add it and also add the number of hours spent in the first instance of this project. Then, just keep adding for each project:

for k in readlineslist:

if k[0] in names:

for z in range(0,names):

if readlineslist[z]==k[0]:

timelist[z]+=k[-1]

....

Get the picture? Of course I write off everything to a text file for billing purposes.

Well, time to hit the road to the real office. Enjoy your day and happy pythoneering!

Automate text processing with string methods

Have you ever had the need to reorganize a text file? Say you have some file that is organized in the following way:

Variable1
20
Variable2
30
....

Say you would like to reorganize that into something more spreadsheet-like, such as

Variable1 20
Variable2 30
---

Easy enough by hand in Excel or OOo calc if there are a couple of values. What if your list contains 20.000 entries? Then a script is your hero. Let's make a command line tool to reorganize such files using Python. We want the script to work like this on the command line:

>> reorganize inputfile.txt outputfile.txt

Therefore, we need to import the sys.argv list (containing command line arguments). Next, we need to use file methods to read the contents of file A into a list that we can operate on. The start of our script is thus:

#!/usr/bin/env python

from sys import argv as filenames

inputfile=open(str(filenames[1]),'r')
inputlist=inputfile.readlines()
inputfile.close()

Now we have a Python list like this:

inputlist=['Variable1\n','20\n','Variable2\n','30\n']

where the '\n' thing means line break. We cannot just write this directly to a new file, we need to strip the '\n' from list elements with even index numbers (indexing starting at zero) and we need to add '\t' to the same list elements to get a tab-separated file easily. An easy way to do this is the following; make two lists, one holding the "names" and one holding the "values". While making these lists, remove all line breaks using the replace method of string objects and replace them with tab characters ('\t'). The code:

ColA=[]; ColB=[];
for k in range(0,len(inputlist)/2):
ColA.append(inputlist[2*k].replace('\n','\t')
ColB.append(inputlist[2*k+1]

Then, simply write to the output file:

outputfile=open(filenames[2],'w')
for k in range(0,len(ColA)):
outputfile.write(ColA[k]+ColB[k])

outputfile.close()

Putting these commands together yields a script that gives you the desired result... (This is a tool I use every day - text processing is so much simpler for a scripter than for the average Excel user). Note that it is possible to do this directly in a spreadsheet using macros. I might consider writing the solution to that one day to. Or not.. :-)

regKPI: register a KPI value in a database or text file

Some days ago I wrote a post on a simple console program. One of the sub-routines in that program was a function called regKPI. We will now dwelve into this function a bit (see original post at A simple console program). If we have a system where we have several KPI's to record, the simplest way to deal with this is the ask the user for a KPI name. Then we know that the answer should be treated as a string. For this we use Python's raw input construct:

def regKPI():

KPIName = raw_input("What is the name of your KPI ?")

---

This way the user does not have to enclose her answer in string delimiters to make the program understand we are talking about character strings. The next question can be "what is the value of your KPI?", and we can take this as a string too. If we are only expecting numbers, we can use just input, and the numeric form is taken directly, but then it has to converted to a string afterwards if it is going to be written into an ASCII file (my favorite "mini" database).

To see how to write the data to a text file, wee the blog post "Writing time data to file". That pretty much sums it up for now. Next time we will look at how we can get KPI data from the "database" and do some statistical analysis on them using Python tools! (Hm... this is getting away from the core concept here of making command line tools - we'll finish this console program thing and return to our true path afterwards! Promise!).

Johan Louwers Personal Blog.: Python dictionary variables

Reading tips: another Python blog:

Johan Louwers Personal Blog.: Python dictionary variables

A simple console program

Say you want to make your own console program. This program should be able of recording data you enter to a text file, and extracting data from the text file to analyze them. One such program can be a tool to register values of different variables you are measuring, such as the number of visitors you have had in your office today. Say you count office visits, and you want to store this in the file under the name "office". Then, you also count how many times during the day your phone rings. This variable is called "phone". You create a program that pops up this menu:

1. Register KPI

2. Create KPI report

3. Quit

>>

When you choose 1, you are prompted for name an value of variable.

The menu is displayed when a function called mainmenu is called. Such a menu can be written as follows in Python:

def mainmenu():

print '1: Register KPI'

print '2: Create KPI report'

print '3: Quit'

menuChoice = raw_input('>> ')

Depending on the input the user gives, we cann call different functions (still within mainmenu):

if menuChoice == '1':

regKPI()

elif menuChoice == '2':

createReport()

elif menuChoice == '3':

print 'Goodbye'

else:

print 'Not a valid menu item. Pleast try again'

main()

There you go, a working menu. Next time: we show what the functions regKPI and createReport look like :-)

Console programs

There are many programs that run in console environments. They are more often seen on Linux than other systems, but that doesn't mean they are not useful on Windows. My *nix favorite console programs are

• alpine: e-mail client for the console (easy to use and with a ton of features)
• midnight commander: file organizer
• vim: text editor
• gnuplot: open source plotting program

These are quite complicated also when it comes to user interfaces, and are very usable programs.

If you are a little less ambitious, making simple console programs with "menus" is quite simple. A database handling program may have the following menu:

MENU:

1. REGISTER NEW FIELD

2. EDIT FIELD VALUES

3. HELP

SELECT MENU ITEM >> ......

Easy enought to use, eh? You want to read the help file, hit 3 on your keyboard followed by enter. Next post will show how to build a quite useful program using a console interface of this type.... (not uncommon in apps from the 1970's and 1980's)!

Avoid the *.py extension when calling scripts under Windows

If you'd like to make your scripts run by typing

myscript args

instead of

myscript.py args

you can wrap the python script inside a *.bat file. Say you want to run myscript.py and it takes two arguments, make the following batch DOS file:

myscript.py %1 %2

The percentage+number things play the role of sys.argv in Python such that command line arguemnts are piped through to the Python script.

Command line tools take command line arguments

You all know them and use them; command line tools with arguments. Especially on linux, this is the way things go, but also in Windows. Think about the dos command dir, and its attributes;

• dir /p: list directory one screen full at the time
• dir /w: list directory in columns,
• ....

On linux you have ls with a bunch of options, for example

ls -al

to show files with owners and permissions. How do we make tools that can take command line arguments in Python?

Fortunately, as the www.python.org site says, Python comes with batteries included. Command line arguments are stored in a list that can be accesed from the sys module. The stdin (command line input) arguments are found using sys.argv. If you use this line:

from sys import argv as inputargs

inputargs will be a list containing text strings with your input arguments. Then, if you have some script like this

from sys import argv as inputargs

myString = 'Hello '+inputargs[1]+', this is '+inputargs[2]+' speaking!'

print myString

The result of running this command:

./myscript.py guest "the owner"

will be:

Hello guest, this is the owner speaking!

Nice to know! We'll put it to use in the next post! Happy hacking :-)

Writing time data to file

In the last post I wrote about a small command line utility for registering hours spent on different tasks. I never got to showing how to write to text files in Python, so let's get down to it now (what's better to do a Saturday evening than hacking away in Python?)!

First, to open a file for editing, you use the following syntax in Python:

myFile=open('filename.txt','attribute')

where attribute is one of the following:

• a - append: appends new text to the end of the file
• w - write: write to file, will overwrite contents already there
• r - read: does only give read access to the file

For our purpose we will use the append attrribute (a). What you create in this way is a file object, that has certain methods available to it depending on the attribute given to the constructor open. The one we will be using is, surprisingly enough, write:

myFile.write(myString)

When you create a file object, the entire file is loaded into your computers working memory (RAM). When you are done working on the file, you should close it to free memory and avoid unforeseen computer trolls to pop out and eat you. You do this by issueing the close method to your file object;

myFile.close()

If you add the three code lines written here to the script outlined in the last post, you will have a working time registering tool!

Time registering command line tool in Python

Often, I want to write a one-liner to a file to register information. I do my time tracking this way, for example. First, you need Python to get data from the command line. Say you want to track number of hours spent on a project at work and get this into a tab-delimited file that you can open in a spreadsheet later (Excel, Calc, etc). You want a utility where you simply write the following at the command prompt: 

regtime projectname hours

and it should automatically make the following line in your "spreadsheet":

date projectname hours

In order to do this, we need to get the current date in Python. Luckily, there is a nice module called time that allows us to do that in a simple fashion. First, import it as follows: 

from  time import strftime as getdate

currentDate=getdate('%Y-%m-%d')

What this does is that it gets the current local time used on the computer and writes a text string with the date:

currentDate='2009-05-22'

We also need to get the command line arguments projectname and hours into Python.

For this we need the list sys.argv.

from sys import argv as arguments

projectname=argv[1]; hours=argv[2];

Ok, lets summarize with a script that takes the two command line arguments and writes the "spreadsheet" line to the screen:

from sys import argv as cla 

from time import strftime as getDate 

myString = getDate('%Y-%m-%d')+'\t'+cla[1]+'\t'+cla[2]+'\n' 

print myString

The result of running this script with arguments "Command line tool" and 1 is shown below:

2009-05-22      Command line tool       1

Next post will show how to write this at the end of a text file. 

Some comments on the above code:

a) the character \t means "insert tab in text string"

b) the character \n means "insert line break"

c) the string '%Y-%m-%d' formats the date to ISO standard

d) adding string elements works in the obvious way: 'foo'+'bar'='foobar'

Write your own command line tools!

As a multi-platform computer user, I always miss some things when going from one platform to antoher. Especially when I go to Windows, I miss some command line tools. One remedy would be Cygwin, but for some reason I don't get the idea of Cygwin. I'd rather use Linux directly then, and I don't need complicated tools. I just like to have some simple possibilities. This blog will contain posts about writing small command line tools in Python that work on Windows (XP), Macs and Linux computers, all using the same scripts, or with very minor modifications. This is kinda my hobby, but it has also made my work day more fun.