Ryan’s Tech Blog

I’ve been able to take a break from my regularly scheduled duties and spend some time working with R.  This is a short log of what I did to get it working.

The main things I’m looking to do is regression modelling from a large dataset I have in postgresql and various stats calculations on some business data I have in SQL Server.  Today I got to the stage in my R learning where I wanted to hook up the databases.

My setup:

• R version 2.12.0 on windows 7
• postgresql 8.4.5 on ubuntu server, requiring SSL
• MS SQL Server 2005 on Windows 2003

R connects to databases via RJDBC, which (surprise) uses JDBC.  You need to download JDBC drivers for each server, and then can load those up inside R.

1. Install RJDBC
   1. Open R
   2. Packages -> Install package(s)
   3. pick a mirror near you
   4. select RJDBC
2. install JDBC driver for MSSQL
   1. I used jtds: http://jtds.sourceforge.net/ (there is also a Microsoft provided driver I didn’t hear about until I was done)
   2. download and unzip
   3. note the path to the jtds jar file (hereafter referred to as $JTDS and the jar filename
   4. open http://jtds.sourceforge.net/faq.html#driverImplementation, which has some magic strings JDBC wants
   5. optional – copy $JTDS/(x64|x86)/SSO/ntlmauth.dll into your %PATH% if you want to use windows authentication with SQL Server
3. install JDBC driver for Postgresql
   1. Download from http://jdbc.postgresql.org/
   2. note the path to the jar file (hereafter referred to as $PG) and the jar file name
   3. open http://jdbc.postgresql.org/documentation/head/load.html, which has some magic strings JDBC wants

Then, to connect with MSSQL:

> library(RJDBC)
> mssql <- JDBC("net.sourceforge.jtds.jdbc.Driver", "$JTDS/jtds-1.2.5.jar", "`")
> testdb <- dbConnect(mssql, "jdbc:jtds:sqlserver://host/dbname")
> typeof(dbGetQuery(testdb, "SELECT whathaveyou FROM whither"))
[1] "list"

And you’re off and running with a list of your results in a list and can do whatever you like.

Now for postgresql+ssl:

> pgsql <- JDBC("org.postgresql.Driver", "$PG/postgresql-9.0-801.jdbc3.jar", "`")
> testdb <- dbConnect(pgsql, "jdbc:postgresql://host/dbname?ssl=true", password="password")
> typeof(dbGetQuery(testdb, "SELECT whathaveyou FROM whither"))
[1] "list"

The connection here has a lot more options, and depends highly on your server’s pg_hba.conf.  It took a little while figure out the “?ssl=true” bit.  Luckily you get pretty descriptive error messages if you can’t connect, and the PostgreSQL JDBC docs are pretty good.

Now to re-learn everything I once knew about regression modeling!

Yesterday I had a need to do some batch processing, making many HTTP calls to crawl a government website to pull down some public data.  I didn’t want to run this during normal hours, because I didn’t want to put a strain on their server.  I had the web crawling sorted out as a function using drakma and cl-ppcre to get the data I wanted.

I wasn’t sure how best to schedule this function to run later that night, and was about to dig into anacron and figure out the right command-line flags to sbcl, when Nathan suggested the obvious:  why not have lisp schedule it?  It was very easy to throw something together using sleep and get-universal-time.  This morning I abstracted that into a helper function using SBCL Threading:

(defun run-later (utime-to-run function)
  (sb-thread:make-thread
   #'(lambda ()
       (sleep (- utime-to-run (get-universal-time)))
       (funcall function))))

Very handy, doesn’t lock my repl, scratches my itch, and easy to write.

I’m getting married in a few weeks, and one of the final tasks was to make some favors for the guests. Heather and I decided on coffee mugs with a cute logo. We quickly came up with the concept of a heart with gears inside, as we’re having a steampunk/victorian themed wedding.  After one very frustrating evening with the gimp (which ended in a crash before I saved the file), I decided attempt two would be procedural, so I fired up my trusty REPL and vecto:

final version

I seriously considered how to calculate the gear size, teeth, and rotation so they mesh perfectly, then gave up and specified them manually via guess-and-check.  The code is at http://paste.lisp.org/display/71465.

I was pretty happy, it ended up being one evening of hacking, with much of that taken rendering test images to get gear placement, rotation, and teeth correct.  I was able to get everything drawing in terms of one variable *r*, so I could scale the size of everything simply.  In retrospect, that could probably have been done just as easily using vecto:scale.  I used vecto:with-graphics-state more than I have before, and that made the drawing operations very straightforward.

I got a little optimistic with the gear-internal generic method, I thought to make many different styles for the inside of the gear, but ended up with only solid and spoked varieties, so had only one specialization for the generic method.

The internal shape of the gear (the number of spokes, or if it’s solid) was determined randomly, and I wrote a quick make-samples function to generate a batch of images.  Then I reviewed those with the fiancee to find a random arrangement we liked.

To pick a suitable font I found all the .ttf files on my computer, then looped through those to make a test logo for each font (the test-font function).  This proved pointless, as Heather knows her fonts and picked one from memory while I was scrolling through hundreds of png files.

The logo is off at the mug printers, and I should be getting a test mug in the mail this week.  Hopefully it will work out and I can offer secret alien technology to all my wedding guests.

The docs on cl-smtp are a little, um, terse, so I figured I’d post a few snippets for google to find:

Sending html email with cl-smtp:

(cl-smtp:send-email
 +mail-server+
 "from-email@example.com"
 "to-email@example.com"
 "Subject"   
 "<html><body>
<p>
Shiny <strong>h</strong><em>t</em><small>m</small>l.
</p>
<p>
</body></html>"
 :extra-headers '(("Content-type" "text/html; charset=\"iso-8859-1\"")))

Sending attachments with cl-smtp:

(cl-smtp:send-email
 +mail-server+
 "from-email@example.com"
 "to-email@example.com"
 "Subject"   
 "see attachement"
 :attachments '("/path/to/attachment"))

Ok, back to my regularly scheduled slog.

I’m finally working on a project that has a public component (besides a login screen), and the lisp has been flowing. I’ve been able to put in probably around 10 hours a week on this thing, and I can’t wait until its more presentable to show off. As part of this, we’re building up our work lisp codebase, and I’ve been keeping my eye out for opportunities to open source different library components, and trying to avoid creating a big ball of mud.

The first real candidate so far was adw-charting, but I think I found the next potentials: adw-yahoo and adw-google. They will be libraries for interfacing with various Yahoo! and Google services. So far I’ve only got code for one service apiece, but there’s plenty of growth potential.

Here’s a sample:

(defun geocode-fight ()
  (let ((adw-yahoo:*api-key* "YAHOO APPID")
	(adw-google:*api-key* "GOOGLE API-KEY")
	(address "5308 SW 75th ter, Gainesville FL, 32608"))
    (list
     (adw-yahoo:latlong address :cache-p nil)
     (adw-google:latlong address :cache-p nil))))

(geocode-fight)
=> (("address" 29.604265 -82.42349 "5308 SW 75th Ter")
(8 29.604923 -82.42338 "5308 SW 75th Terrace, Gainesville, FL 32608, USA"))

Yahoo and Google give back different some different data, but so far Yahoo’s coordinates seem more accurate. One thing I noticed was that Google’s geocoding service gives a different lat/long than what maps.google.com displays, which I thought was a little sneaky. I’ve also got some code to help construct google map widgets, using the homegrown html template system and parenscript, but I think much of that will end up getting stripped out, as it depends on the in-house libraries that are likely completely useless to anyone but us.

Still lots to do before either of those libs would be ready for a cl-net request, but it’s on the plate. After building so much on top of so many great free libraries, the need to contribute something back is very strong.

Some more progress on the charting library today:

• cleaned up the API a little bit, opting for some nicer make-foo function instead of requiring calls to make-instance:
  • make-series label data &key color
  • make-axis label &key control-string draw-gridlines-p label-formatter
    • control-string: a format-compatible control string, and supplying it sets the label-formatter for the axis
    • label-formatter: a function of 1 argument, returns a string used as an axis label
  • make-line-chart width height &key series y-axis x-axis background
• The lightened portion of the graph is now the average of the background color and pure white
• The labels on the axes are optional, and the graph adjusts to fill as big an area as possible
• The distance between x-axis labes is calculated based on actual string-widths, so hopefully they will space themselves out reasonably under most conditions

Some examples:

This demonstrates using a control-string and a custom function for axis labels:

  (render-chart
   (make-line-chart 400 300
		    :background '(.7 .7 .7)
		    :series (list (make-series "SeriesA"
					       '((-1 -2) (0 4) (1 5) (4 6) (5 -3)))
				  (make-series "SeriesB"
					       '((-1 4) (0 -2) (1 6) (5 -2) (6 5))
					       :color '(.3 .7 .9))
				  (make-series "SeriesC"
					       '((-1 0) (0 3) (1 1) (2 5) (4 -6))))
		    :y-axis (make-axis "widgets"
				       :control-string "~$")
		    :x-axis (make-axis "time"
				       :label-formatter #'months-from-now->mm/yy))
   "line-chart-with-axis-labels.png")

This one changes the background color, removes the axis labels, and disabled grid lines for the x-axis:

(render-chart
   (make-line-chart 400 300
		    :background '(.7 .5 .7)
		    :series (list (make-series "SeriesA"
					       '((-1 -2) (0 4) (1 5) (4 6) (5 -3)))
				  (make-series "SeriesB"
					       '((-1 4) (0 -2) (1 6) (5 -2) (6 5))
					       :color '(.3 .7 .9))
				  (make-series "SeriesC"
					       '((-1 0) (0 3) (1 1) (2 5) (4 -6))))
		    :y-axis (make-axis nil
				       :control-string "~$")
		    :x-axis (make-axis nil
				       :draw-gridlines-p nil
				       :label-formatter #'months-from-now->mm/yy))
   "line-chart-with-axis-labels.png")

The internals are still messy, but I’ve gone through and was able to simplify some things here and there, here’s a few of the change’s I’ve made to make my code shorter and more readable:

• replacing (destructuring-bind (x y) (foo) (bar x y) with (apply #’bar (foo))
• adding a macrolet to concentrate some very similar behavior
• using loop to count up and down at the same time (for drawing labels up/down the y axis and left/right on the x axis)
• using (dolist (item lst) …) instead of (mapc #’(lambda(item) …) lst)
• using loop‘s maximizing / minimizing features
• using multiple-value-list and nth to get specific values from some function results without compiler warnings

I’ve been trying to put my tongue on why exactly I like lisp so much more than other languages, and working through this made it pretty clear: I have much better tools for reducing accidental complexity than in other languages, and can get the code focusing on what I’m trying to do, not how my compiler needs me to do it. The next time I get a free afternoon, I’m going to make an imperative API, similar to vecto’s:

(with-line-chart (400 300 :background '(.7 .7 .7))
    (add-series "SeriesA"
		'((-1 -2) (0 4) (1 5) (4 6) (5 -3)))
    (add-series "SeriesB"
		'((-1 4) (0 -2) (1 6) (5 -2) (6 5))
		:color '(.3 .7 .9))
    (add-series "SeriesC"
		'((-1 0) (0 3) (1 1) (2 5) (4 -6)))
    (set-axis :y "widgets" :control-string "~$")
    (set-axis :x "time" :label-formatter #'months-from-now->mm/yy)
    (save-file "line-chart-with-axis-labels.png")

That’s pretty damn readable.

updated: fixed syntax error in the imperative sample, and got the imperative bit actually working.

It now does line charts and better pie charts, and has had a lot of bugs removed since the last post. We did some code review on it at work, brought some of the spaghetti under control, and included the library in an intranet app.

Here’s a sample line graph:

The (somewhat verbose) code for this:

(defun line-chart-sample ()  
  "draws a simple line chart"
  (let* ((seriesA (make-instance 'series
				 :label "SeriesA"
					;data expressed as a list (x y) pairs
				 :data '((-1 -2) (0 4) (1 5) (4 6) (5 -3))))
	 (seriesB (make-instance 'series
				 :label "SeriesB"
				 :data '((-1 4) (0 -2) (1 6) (5 -2) (6 5))))
	 (chart (make-instance 'line-chart
			       :width 400
			       :background '(.7 .7 .7)
			       :series (list seriesA seriesB))))
    (render-chart chart "line-chart-sample.png")))

I’ll probably make some easier syntax for creating series, similar to the one I made for pie slices, but for now the CLOS approach has been working decently. My data is coming from clsql, so the code ends up more concise than this example.

There are a few spots internally where it gets a little nasty, needing to pass many things around to many functions. I may end up changing the style to use “with-line-chart” functions with some global variables tracking the current chart, but I’m not sure how that would work in a multi-threaded environment. Hell, I’m not sure if my existing code will work in a multi-threaded environment. The load on this intranet app is very low, so I may be lucky enough to get away with it.

Gonna try to be quick about this, as there’s chicken on the grill. I took a break from Team Fortress 2 to play some more with Vecto, another fine library from Xach. I started another post about last weekend’s lisp playing, but this one got finished first.

I’ve been tracking my fuel efficiency for a few weeks, noting the date, gallons purchased, price per gallon, and miles since last refill. You can’t control what you don’t measure. I had a ruby script keeping up with these as YAML and then giving me some handy stats, but I’m trying to gain some levels in lisp, so I decided to fire up slime and make some graphs.

The graph isn’t pretty, but here’s my miles per gallon:

As usual, I pulled up my favorite lisp tutorial and #lisp, and after a few hours, I had a graph. Given my logging fetish, I imagine I’ll be growing the graphing capabilities in the future.

Right now, the implementation is a little weird. I have an class for a fuel entry, and then convert the date to a universal-time using net.telent.date. Then in my graphing function I use the universal time for the x values, and any number of functions for the y values (defaulting to #’mpg-of). In order to get the graph to take up most of the canvas, I then normalize the x and y values, so the minimum is 0 and the maximum is 1, and then scale the number to fit the canvas size before drawing.

I still consider myself a lisp beginner, and there are some bits I’m sure could be more efficient, but it works for now. Those interested in the code can get it here: fuelman.tar.gz, and any feedback in comments is much appreciated.

Update: Played with it a lot more, simplified my normalization crap a lot using loop instead of mapcar, and pulled it into a couple of objects: graph, graph-series, and fuel-graph-series.  Also linked the source above as a proper .tar.gz file, after reading how to do that in wordpress.  The aesthetics are still terrible, but it’s getting there:

I’m working on a simple command-line app, and have the need to collect a username and password. I don’t want the password to be printed on the screen as they type, but System.Console.ReadLine doesn’t seem to have any option to mask the input before it’s echoed back to the console.
There are a couple ways I found to resolve this, the easy way, and the harder way with better UI.

Easy way:
[csharp]
ConsoleColor oldFore = Console.ForegroundColor;
Console.ForegroundColor = Console.BackgroundColor;
string password = Console.ReadLine();
Console.ForegroundColor = oldFore;
[/csharp]

This basically hides the echoed input by setting the text color to be the same as the background color. This is a little weird because your users can’t see any indication that they’ve entered any information.

The following function catches the input, and then echoes “*” instead. There’s some rudimentary handling of backspace, but no other special keys (end, delete, arrow keys, etc) are handled properly. For now it’s good enough for my use:
[csharp]
public static string ReadPassword() {
Stack passbits = new Stack();
//keep reading
for (ConsoleKeyInfo cki = Console.ReadKey(true); cki.Key != ConsoleKey.Enter; cki = Console.ReadKey(true)) {
if (cki.Key == ConsoleKey.Backspace) {
//rollback the cursor and write a space so it looks backspaced to the user
Console.SetCursorPosition(Console.CursorLeft – 1, Console.CursorTop);
Console.Write(” “);
Console.SetCursorPosition(Console.CursorLeft – 1, Console.CursorTop);
passbits.Pop();
}
else {
Console.Write(“*”);
passbits.Push(cki.KeyChar.ToString());
}
}
string[] pass = passbits.ToArray();
Array.Reverse(pass);
return string.Join(string.Empty, pass);
}
[/csharp]
A bit messy and bug-ridden, but for now it’s good enough for my purposes. When C# 3 comes out, I might be able to add things like this to the Console class using extension methods. Maybe a community-driven FrameworkPlus library will get some momentum, and we can all ditch our home-grown Utils libraries and reap the benefits of each other’s work.

I was recently implementing some custom events, and found a couple of good (if old) articles describing how to do this efficiently using EventHandlerList:

• Manage many events easily with EventHandlerList
• Objects with dense events, but sparse usage can benefit from custom event storage.

Those articles go into why it’s nicer to deal with one EventHandlerList instead of many seperate EventHandlers, so read those for more information. For the lazy, here’s some code showing how you’re supposed to use these things:
[csharp]
public class MyClass {
private EventHandlerList Events = new EventHandlerList();

public event EventHandler MyEvent {
add { Events.AddHandler(“MyEvent”, value); }
remove { Events.RemoveHandler(“MyEvent”, value); }
}

public event EventHandler MyOtherEvent {
add { Events.AddHandler(“MyOtherEvent”, value); }
remove { Events.RemoveHandler(“MyOtherEvent”, value); }
}

protected void OnMyEvent(object sender, EventArgs e) {
EventHandler handler = (EventHandler) Events["MyEvent"];
if (handler != null) {
handler(sender, e);
}
}

protected void OnMyOtherEvent(object sender, EventArgs e) {
EventHandler handler = (EventHandler) Events["MyOtherEvent"];
if (handler != null) {
handler(sender, e);
}
}
}
[/csharp]

Pretty straightforward stuff. When you add an event handler to the list, you associate it with a key, and then when its time to trigger the events, you look for any handlers under the same key. The other day I was putting together something similar, and ran into some unexpected behavior with the keys. I had started by refactoring the magic strings into an enum:
[csharp]
protected enum MyEvents {
MyEvent,
MyOtherEvent
}
[/csharp]
and replaced all the strings with members of that enum. I figured this would work just fine, but the change caused my unit test to fail. Upon debugging, the EventHandlerList was always returning null in my On*Event calls. After some more testing, the pattern became apparent: value types don’t work as keys. This was somewhat unexpected, as I’ve used enums like this in Hashtables all over the place before. After doing a little Reflectoring, the actual search for the key comes down to traversing a linked list with a simple equality test, something like this:
[csharp]
while (head != null)
{
if (head.key == key)
{
return head;
}
head = head.next;
}
[/csharp]
The culprit ends up being C#’s auto-boxing. The key is stored as an object, so my value types are being boxed on the way in, and therefore == is comparing object identity, not the object values. If EventHandlerList used head.key.Equals(key), everything would have worked how I expected. The solution to rid myself of magic strings now becomes using static objects as my keys, so the object identities will match:
[csharp]
private static readonly object MyEventKey = new object();
private static readonly object MyOtherEventKey = new object();
[/csharp]
That pattern reminds me a lot of enums in Java before it got a enum keyword, which came on the heels of C#’s nice solution to the enumerated type problem. It’d be nice if I could use enums for their intended purpose, but cases like this make me a bit wary. Where else in the .NET framework am I going to find object identity equality where I expect to find object value equality? Is there some rational explanation for this, or is this just a bug?