Category Archives: Linux
Announcing DateRangeParser: Parse strings like “27th-29th June 2010″
In a project recently I was struggling to find a way to parse strings that contain a date range, for example:
- 27th-29th June 2010
- Tuesday 29 May -> Sat 2 June 2012
- From 27th to 29th March 1999
None of the Python modules I investigated (including parsedatetime) seemed to be able to cope with the range of strings that I had to deal with. I investigated patching parsedatetime to allow it to do what I wanted, but I found it very hard to get into the code. So, I thought, why not write my own…
So I did, and I’ve released it under the LGPL and you can install it right now by running:
pip install daterangeparser
Or you can visit the DateRangeParser PyPI page to download it manually, read the documentation, or hack on the code.
The current version will parse a wide range of formats (see the examples in the documentation) and will deal with individual dates as well as date ranges. The API is very simple – just import the parse method and run it, giving the date range string as an argument. For example:
from daterangeparser import parse
print parse("14th-19th Feb 2010")
This will produce an output tuple with two datetime objects in it: the start and end date of the range you gave.
The parser is built using PyParsing – a great Python parsing framework that I have found very easy to get to grips with. It is incredibly powerful, very easy to use, and really shows how limited regular expressions can be! Now that I’ve done this I have an urge to use PyParsing to write parsers for all of the horrible scientific data formats that I have to deal with in my PhD….watch this space!
Reading data from instruments via RS-232 simply in Linux
As part of my research I do a fair amount of data collection in the field. Some of the instruments I use are very modern and connect to a computer via USB, interacting with custom-written client software which allows such luxuries as timed logging, triggered logging 
and local calibration. However, a number of the instruments are older and don’t have computer-based logging capability, requiring you to log data to their internal memory and then download it later.
This is often perfectly satisfactory, but timing can be an issue. For example, when taking measurements using a number of instruments it is often important to make sure that measurements are taken at the same time. For example, if spectral measurements are being taken and other instruments (for example sunshine sensors, like the sensor shown below) are being used to gather data which can then be used to atmospherically correct the spectra, then it is very important to ensure that measurements are taken at the same time. This is particularly a problem in areas of fast changing weather like the UK, where sky conditions can change very quickly.
A tool called SJinn allows you to send simple strings over a RS-232 (standard serial port) connection and then obtain data sent back by the instruments. One of the examples given by SJinn is the following:
rs232 -b600 -p7n2 -s"\n" -r16
This sends a newline character over the serial port (at 600 baud with 7 data bits and 2 stop bits) and then returns the next 16 characters send on the line. In this case, it would provide the voltage measured by a digital voltmeter. As this is simply a command-line tool, it is very easy to combine into scripts, and thus use to collect timed measurements (eg. via the use of the cron daemon). I have used similar techniques to obtain measurements from the sunshine sensor shown above – a script for which will be available on my website soon.
Amazing software you haven’t heard of
Every so often, on my travels around the internet, I come across a piece of software which is so great that I wonder why on earth I haven’t heard of it before. The software listed below falls into this category, and hopefully by posting the list here I will allow more people to find them.
AeroFS
This is an online file-synchronisation service similar to Dropbox but with one key difference: nothing is stored on a cloud server unless you specify that it should be. That is, the synchronisation takes place through a securely encrypted tunnel between your computers running AeroFS, and is never stored in the cloud. This has a number of benefits: it means you can store as much as you want on your AeroFS drive without having to pay for cloud storage, and it means that data is not stored on third party computers (essential for some business applications). It is cross-platform (Windows, Linux, Mac) and free – what more could you want?
Caffeine
This simple app does one simple thing, but is invaluable. Do you ever find that your MacBook screen’s backlight goes off while you’re busy watching a film, showing your family photos, or busy watching a process complete. By clicking the coffee cup icon that Caffeine puts in your menu bar you can stop the screen backlight from switching off. Simply click the icon again to get it back to normal.
Max
We’ve all done it: suddenly needed to convert an audio file and googled “Convert from X to Y” and found a huge list of ad-riddled pages explaining how to do it if you buy their ghastly shareware software. Although I sometimes like to stick to good-old command-line tools like ffmpeg, I quite like finding a nice GUI tool to do this. Max allows you both to rip CDs (through a variety of methods) and convert audio files that you already have, all through a nice GUI interface, with no dependencies on other software. Unfortunately it’s Mac only.
Evom
Similar to Max, but for video – this program will convert any video files you have to other formats, and download YouTube videos to any format you want. It’ll even let you convert files into just the right format for playing on various hand-held devices (iPods, iPads, mobile phones etc).
DTerm
I’ve mentioned DTerm before on my blog, and I really can’t live without it on my Mac now. It allows you to quickly open a simple command prompt in any directory, and execute a command there (with full output shown), or switch immediately to a terminal focussed on that folder, ready to do any other processing you might need. It does full command-line completion, and I haven’t yet found a command that won’t work in DTerm’s terminal.
How to: Set up a simple service to run in the background on a Linux machine (using daemontools)
I have just set up a new home server (a review of which will be coming soon) and have been installing various programs that I want to run on it. A number of these are servers, such as sshd, apache, samba etc. All of these have fairly easy installs under Debian and will automatically run at startup, and can be controlled by /etc/init.d scripts.
However, I also have a number of other programs I want to run as services, constantly in the background, which don’t come with nice Debian init.d scripts. After asking a question on SuperUser I found that one fairly simple solution would be to use a set of tools called daemontools. These tools provide a simple way of defining services which run constantly, and can be controlled by an administrator (in a similar way to /etc/init.d services). daemontools seem to be designed very well, and are quite easy to use, but the documentation on the website seems to lack a simple quickstart guide…so I thought I’d write my own.
At this point I should mention that I have only been using daemontools for a few hours, so I could be completely wrong about anything I say below. These instructions will be for Debian, but should be fairly easily to use with other distros (the only bit that will be significantly different is exactly how to install it with the package manager). Anyway, proceed at your own risk!
- Install daemontools In debian you will need to run apt-get install daemontools daemontools-run (both packages are important – I didn’t install the latter package and it caused me lots of frustration). This will install the tools themselves and also add the required lines to startup files to ensure that all of the required daemontools services start when the machine boots.
- Create a service directory You will probably find that the installer has created a /service directory for you. If it hasn’t then create one yourself. Then create a directory under that directory for each service that you want to run. Here we will be creating a test service, so create a directory called test. Run chmod 1755 on this directory.
- Create the service run file
daemontools needs to know what command(s) you need to run for this service, and these commands should be put inside a shell script called run in the service directory. For example, the file could contain:
#!/bin/sh echo Running service exec some-command-here
- Finished! That should be all you need to get the service running. You should probably restart the machine now as that will ensure that all of the daemontools monitoring services have started correctly. Once the machine has started the new service should have started running. If it crashes or ends for some reason it will restart after one second. Any new services you add (which you can do exactly as above) should start within five seconds. You can use the svc command to control the services you have created (see man svc for details)