Here’s how it works in a nutshell:

1. From javascript I pass to the server the lat/lng region currently shown on the google map, and what size heat map to generate, in pixels.
2. lisp pulls weights from my database within the given lat/lng region
3. lisp iterates over the db results, mapping lat/lng to x/y coordinates for the final heat map image
4. lisp uses the list of mapped (x y weight) to draw the heat map in png
5. javascript throws the png on top of the google map

I tried a few things based upon the comments I got back from the helpful lisp community.

• used zpng to get direct pixel access, and calculated each pixel’s color using a weighted average of nearby points using distance.  This didn’t produce good images, and was pretty slow.
• used zpng to get direct pixel access, and calculated each pixel’s color using the gravity formula against nearby points.  This didn’t produce good images, and was very slow.

I did some more research and learned about the Generic Mapping Tools and bicubic interpolation. The GMT is a set of C programs, similar to the Imagemagick suite.  GMT showed one way to draw heat maps in the Image Presentations tutorial.  It spoke of gridded data sets, and that gave me one more vecto-based idea: split the desired heat-map into a grid and color each square in the grid based upon an average of the weights mapped in that square.  This is a neat effect, but not what I was going for:

This is reasonably fast, taking about 1 second on my dev server.  To quickly find what weights belong in which grid square, I make a spatial index of all the weights, using an r-tree from the spatial-trees library.

The next method I tried was to use interpolation to get a smooth look.  I found Cyrus Harmon‘s ch-image library supports image interpolation, and got to it.  As Patrick Stein noted elsewhere, ch-image isn’t easy to install.  It’s not asdf-installable, and the project page doesn’t list all its dependencies.  For future reference, here’s what I think I needed to install it:

(asdf-install:install "http://cyrusharmon.org/static/releases/ch-asdf_0.2.14.tar.gz")
(asdf-install:install "http://cyrusharmon.org/static/releases/ch-util_0.3.10.tar.gz")
(asdf-install:install "http://cyrusharmon.org/static/releases/smarkup_0.4.2.tar.gz")
(asdf-install:install "http://mirror.its.uidaho.edu/pub/savannah/cl-bibtex/cl-bibtex-1.0.1.tar.gz")
(asdf-install:install "http://cyrusharmon.org/static/releases/clem_0.4.1.tar.gz")
(asdf-install:install "http://cyrusharmon.org/static/releases/ch-image_0.4.1.tar.gz")

Armed with ch-image, now the drawing process becomes:

1. draw a small image, coloring pixels based upon weights
2. enlarge the small image with interpolation

The first step is very similar to the code I wrote to make the grid version above.   Instead of drawing a rectangle, I draw a pixel using ch-image’s pixel access functions.  This was a little weird because ch-image’s coordinate system has 0,0 at the top left of the image.  I’m still not sure how to best choose the size of this smaller image, but ultimately it should depend on my data.  For now I just have it hard-coded be 20x smaller than the desired size:

Yep, that’s pretty small.  Applying a transform to scale it up to the desired size using bilinear interpolation yields:

It looks pretty good and takes about a half-second to draw.  If you click into the larger version, you can see some discontinuities in there, which is a well-known result of bilinear interpolation.  However, based upon other graphics I’ve seen, what I really want is bicubic interpolation.  Luckily, ch-image has this built in:

Oops, maybe not so luckily.  I can certainly see the kinds of look I’m wanting in all the garbled stuff, but ch-image is freaking out somewhere there.

Bilinear it is!  Here’s a screenshot of the overlay in place on the map:

It’s pretty fast, and looks pretty nice, and is fairly close to the look I wanted.  I probably still have some off-by-one errors somewhere, and need to check the git repos for the ch-* libs to see if there might be newer versions than the tarballs I installed.  I still count this as great progress for 5 hours of coding and research.  Huzzah for the much-maligned lisp libraries!

There are a few neat things going on here.  I’ve got a url handler in my lisp that looks to the query string for lat-lng bounds, image size, and some other variables to generate a PNG file.   I pass that URL to a Google Maps API GGroundOverlay to put the image onto the map.  Add some javascript event glue and I can do cool things like automatically regenerate the heat map overlay when you zoom/pan the map around, and display an animated heat map showing consumption over the course of the year.  There’s still a lot of UI interaction to sort out, but I think it’s a nice approach.

The heat map itself is generated using Vecto, and I think I’m doing it wrong.  I jump through some hoops to map lat-lng to image pixel coordinates, pull from the database, and end up with a list of (x y weight) tuples, with the weight being a number between 0.0 and 1.0 representing the relative consumption of the home that should be at pixel x,y in the result image.  Then I start painting, which is where I think I should be doing more math.  For each point, I pick a color between green and red based on the weight, using the handy cl-colors library to interpolate:

(defun find-color (percent)
  (if (> .5 percent)
      (cl-colors:rgb-combination cl-colors:+green+ cl-colors:+yellow+ (* 2 percent))
      (cl-colors:rgb-combination cl-colors:+yellow+ cl-colors:+red+ (* (- percent .5) 2))))

I actually have to go from green->yellow, then yellow->red, with some goofy adjustments to the percent to make the interpolation work out.  Once I have that, then I have my color, and my pixel, so I can start drawing.  To get a smoother look, for each point I draw concentric circles with different radius and opacity, so each individual data point is rendered like this:

This is enlarged to show some of the blockiness, it ends up looking pretty nice when they are small.  Here’s the actual function:

(defun draw-point (x y color max-radius)
  (iterate (for r from max-radius downto 1 by (max 2 (round (/ max-radius 6))))
	   (for alpha = (/ 1 r))
	   (vecto:set-rgba-fill (cl-colors:red color)
				(cl-colors:green color)
				(cl-colors:blue color)
				alpha)
	   (vecto:centered-circle-path x y r)
	   (vecto:fill-path)))

Max-radius determines how large the largest circle is, and is calculated based on how many points I’m drawing.

There are a few drawbacks to this approach.  First, it’s slow.  Drawing operations aren’t exactly cheap, especially when messing with alpha channels.  It takes me around 5s for 578 data points, which is fine for offline tasks, but on a web-app it needs to be super zippy or you fickle internet folk will close the tab. I also want it to be easy to show animations, so generating a bunch of them quickly would be nice.  The time spent increases fairly linearly with data points, and I’d like to be able to render heat maps for large areas with tens of thousands of data points.  Profiling shows practically all of my time and bytes consed are spent in the draw-point function. UPDATE: after more profiling, vecto:fill-path is most of my time, which makes sense.

Second, I have to be really careful to draw these points from lowest weight to highest weight, because I want red dots to be painted on top of green dots.  It seems like I should decide what color each pixel should be, then draw it once, rather then accumulating the right color in the image canvas.  Right now there’s also some bug with drawing lots of data points, I just get a big green image, when I would expect some reds or yellows.

Another issue is for apartments I have coordinates for the apartment complex, but not each individual unit.  This makes some funny results, like the big orange blob on the right side of the screenshot above where I’ve painted a few dozen points on top of each other.

I did some googling on heat-map algorithms, and found some actionscript and java code, but the actionscript was using a similar approach and the java was incomprehensible.  I think I’ll try making a big array for the result image, and calculating an average weight for each pixel, then loop through that and draw once.  I’m also going to try calculating the weights using magnetic field strength or gravity math.  I think that approach will end up faster, look nicer, and should be a fun problem.

final version

I seriously considered how to calculate the gear size, teeth, and rotation so they mesh perfectly, then gave up and specified them manually via guess-and-check.  The code is at http://paste.lisp.org/display/71465.

I was pretty happy, it ended up being one evening of hacking, with much of that taken rendering test images to get gear placement, rotation, and teeth correct.  I was able to get everything drawing in terms of one variable *r*, so I could scale the size of everything simply.  In retrospect, that could probably have been done just as easily using vecto:scale.  I used vecto:with-graphics-state more than I have before, and that made the drawing operations very straightforward.

I got a little optimistic with the gear-internal generic method, I thought to make many different styles for the inside of the gear, but ended up with only solid and spoked varieties, so had only one specialization for the generic method.

The internal shape of the gear (the number of spokes, or if it’s solid) was determined randomly, and I wrote a quick make-samples function to generate a batch of images.  Then I reviewed those with the fiancee to find a random arrangement we liked.

To pick a suitable font I found all the .ttf files on my computer, then looped through those to make a test logo for each font (the test-font function).  This proved pointless, as Heather knows her fonts and picked one from memory while I was scrolling through hundreds of png files.

The logo is off at the mug printers, and I should be getting a test mug in the mail this week.  Hopefully it will work out and I can offer secret alien technology to all my wedding guests.

• charts are limited to 300,000 pixels, which is smaller than it sounds
• because all the data needs to be passed on a URL, it chokes on large datasets.  This can be helped by using alternate encodings for values, but that looks like a pain in the ass (basically have to map the arbitrary lisp data evenly onto 0-64 or 0-4096, then encode as ASCII, I predict rounding troubles)
• it doesn’t automatically scale to fit your size.  For example, a bar chart uses 20px wide bars by default.  If you have too many bars they just wander off the right of the canvas, and you need to manually adjust the bar width to get things to fit
• some of the labels / titles get cut off

The google backend certainly has it’s uses, but I’m going to revive the vecto backend, which gives complete control.  As a start, I’ve split the project into a 3 asdf systems:

• adw-charting: has some utils and common code, declares the :adw-charting package
• adw-charting-vecto: adds/exports functions for making vecto charts in the :adw-charting package
• adw-charting-google: adds/exports functions for making google chart URLs into in the :adw-charting package

In this setup, you’ll load adw-charting-google.asd or adw-charting-vecto.asd, and then all the dependancies will sort themselves out, and you don’t have anything loaded you don’t need.

I looked at asdf-system-connections but didn’t see how it would help, so copied cl-xmpp‘s multiple asd file scheme.  This is currently just in the darcs repo, and may have some bugs.

I’ve also started an actual adw-charting todo list, as part of a greater effort to be more organized in my life, comments welcome here.

• cleaned up the API a little bit, opting for some nicer make-foo function instead of requiring calls to make-instance:
  • make-series label data &key color
  • make-axis label &key control-string draw-gridlines-p label-formatter
    • control-string: a format-compatible control string, and supplying it sets the label-formatter for the axis
    • label-formatter: a function of 1 argument, returns a string used as an axis label
  • make-line-chart width height &key series y-axis x-axis background
• The lightened portion of the graph is now the average of the background color and pure white
• The labels on the axes are optional, and the graph adjusts to fill as big an area as possible
• The distance between x-axis labes is calculated based on actual string-widths, so hopefully they will space themselves out reasonably under most conditions

Some examples:

This demonstrates using a control-string and a custom function for axis labels:

  (render-chart
   (make-line-chart 400 300
		    :background '(.7 .7 .7)
		    :series (list (make-series "SeriesA"
					       '((-1 -2) (0 4) (1 5) (4 6) (5 -3)))
				  (make-series "SeriesB"
					       '((-1 4) (0 -2) (1 6) (5 -2) (6 5))
					       :color '(.3 .7 .9))
				  (make-series "SeriesC"
					       '((-1 0) (0 3) (1 1) (2 5) (4 -6))))
		    :y-axis (make-axis "widgets"
				       :control-string "~$")
		    :x-axis (make-axis "time"
				       :label-formatter #'months-from-now->mm/yy))
   "line-chart-with-axis-labels.png")

This one changes the background color, removes the axis labels, and disabled grid lines for the x-axis:

(render-chart
   (make-line-chart 400 300
		    :background '(.7 .5 .7)
		    :series (list (make-series "SeriesA"
					       '((-1 -2) (0 4) (1 5) (4 6) (5 -3)))
				  (make-series "SeriesB"
					       '((-1 4) (0 -2) (1 6) (5 -2) (6 5))
					       :color '(.3 .7 .9))
				  (make-series "SeriesC"
					       '((-1 0) (0 3) (1 1) (2 5) (4 -6))))
		    :y-axis (make-axis nil
				       :control-string "~$")
		    :x-axis (make-axis nil
				       :draw-gridlines-p nil
				       :label-formatter #'months-from-now->mm/yy))
   "line-chart-with-axis-labels.png")

The internals are still messy, but I’ve gone through and was able to simplify some things here and there, here’s a few of the change’s I’ve made to make my code shorter and more readable:

• replacing (destructuring-bind (x y) (foo) (bar x y) with (apply #’bar (foo))
• adding a macrolet to concentrate some very similar behavior
• using loop to count up and down at the same time (for drawing labels up/down the y axis and left/right on the x axis)
• using (dolist (item lst) …) instead of (mapc #’(lambda(item) …) lst)
• using loop‘s maximizing / minimizing features
• using multiple-value-list and nth to get specific values from some function results without compiler warnings

I’ve been trying to put my tongue on why exactly I like lisp so much more than other languages, and working through this made it pretty clear: I have much better tools for reducing accidental complexity than in other languages, and can get the code focusing on what I’m trying to do, not how my compiler needs me to do it. The next time I get a free afternoon, I’m going to make an imperative API, similar to vecto’s:

(with-line-chart (400 300 :background '(.7 .7 .7))
    (add-series "SeriesA"
		'((-1 -2) (0 4) (1 5) (4 6) (5 -3)))
    (add-series "SeriesB"
		'((-1 4) (0 -2) (1 6) (5 -2) (6 5))
		:color '(.3 .7 .9))
    (add-series "SeriesC"
		'((-1 0) (0 3) (1 1) (2 5) (4 -6)))
    (set-axis :y "widgets" :control-string "~$")
    (set-axis :x "time" :label-formatter #'months-from-now->mm/yy)
    (save-file "line-chart-with-axis-labels.png")

That’s pretty damn readable.

updated: fixed syntax error in the imperative sample, and got the imperative bit actually working.

This is the start of axes. The internals are getting very, very messy, but I really like the result. Here’s the code used to create this:

(defun line-chart-with-axis-labels ()
  "draws a simple line chart"
  (let* ((seriesA (make-instance 'series
				 :label "SeriesA"
					;data expressed as a list (x y) pairs
				 :data '((-1 -2) (0 4) (1 5) (4 6) (5 -3))))
	 (seriesB (make-instance 'series
				 :label "SeriesB"
				 :data '((-1 4) (0 -2) (1 6) (5 -2) (6 5))))
	 (y-axis (make-instance 'axis
				:label "widgets"
				:label-formatter #'(lambda (x-val)
						     (format nil "~$" x-val))))
	 (chart (make-instance 'line-chart
			       :width 400
			       :height 300
			       :background '(.7 .7 .7)
			       :series (list seriesA seriesB)
			       :y-axis y-axis)))
    (render-chart chart "line-chart-with-axis-labels.png")))

I predict adding X axes will be very similar to adding the Y axis, and then I’ll be looking more seriously at spending time to make good docs and releasing this back to the community. The API is getting a little dense, too, but that should be pretty easy to simplify, and I have a few ideas on how to do that.

Here’s a sample line graph:

The (somewhat verbose) code for this:

(defun line-chart-sample ()
  "draws a simple line chart"
  (let* ((seriesA (make-instance 'series
				 :label "SeriesA"
					;data expressed as a list (x y) pairs
				 :data '((-1 -2) (0 4) (1 5) (4 6) (5 -3))))
	 (seriesB (make-instance 'series
				 :label "SeriesB"
				 :data '((-1 4) (0 -2) (1 6) (5 -2) (6 5))))
	 (chart (make-instance 'line-chart
			       :width 400
			       :background '(.7 .7 .7)
			       :series (list seriesA seriesB))))
    (render-chart chart "line-chart-sample.png")))

I’ll probably make some easier syntax for creating series, similar to the one I made for pie slices, but for now the CLOS approach has been working decently. My data is coming from clsql, so the code ends up more concise than this example.

There are a few spots internally where it gets a little nasty, needing to pass many things around to many functions. I may end up changing the style to use “with-line-chart” functions with some global variables tracking the current chart, but I’m not sure how that would work in a multi-threaded environment. Hell, I’m not sure if my existing code will work in a multi-threaded environment. The load on this intranet app is very low, so I may be lucky enough to get away with it.

1. homegrown SVG renderer (javascript)
2. homegrown png renderer (C#)
3. Gruff (ruby)
4. some graphing capability from Dojo (javascript)
5. cl-plplot

I’ve never been happy with any of these, for a few reasons:

1. SVG support for IE is a pain
2. C# requires a million objects to do anything
3. Gruff had good defaults, but had some annoying bugs, and I ended up jumping through a lot of hoops to get some basic aggregation going.  I looked at monkey-patching gruff, but that seemed like a really bad idea
4. The dojo charting was slow, and the setup was verbose (this may have changed)
5. cl-plplot output wasn’t very attractive. Great for scientific usages, but for business apps it didn’t have enough rounded edges.

I want something that’s concise, has good defaults, rendered to PNG, and is easy to use from lisp. So today I tried rolling my own, and I’m pretty happy with the results:

The code to generate this graph is pretty concise, too:

(defun pie-chart-sample ()
  "draws a simple pie chart, rending to pie-chart-sample.png"
  (render-chart
   (make-instance 'pie-chart
		  :data-items (make-items '(40
					    (10 "baz")
					    (60 "bar" (.5 .5 .5)))))
   "pie-chart-sample.png"))

This demonstrates a few things:

1. data can be specified as value, a value/label, or value/label/color
2. it automatically choses colors if unspecified
3. the labels are automatically positioned off to the side

There’s still some places to cleanup the API, I don’t like having make-items in there, and a lot of the behind the scenes is messy as all hell.

I started by looking at the source for Paragent, a lisp-based IT management tool. I first saw really attractive graphs coming from lisp there, and it impressed me a lot. It was building on cl-vectors, salza, and zpb-ttf to make png files. There was a lot of weird stuff going on in the paragent code, so I instead opted to port my old javascript code to lisp, using vecto for the rendering.

I ran into a brick wall pretty early when calculating the wedges, as I’ve forgotten just about all the math I ever knew (had to look up the distance formula, for christsakes). The main problem was figuring out how to draw the curved part. I spent a long time staring at Bézier curves before giving up and taking a different approach. I made a circular clip-path (basically a mask) for the chart, and drew big enough wedges to fill the circle, side-stepping my mathematical shortcomings. There’s some goofy code to account for large slices, but it worked for all values I tested.

I did a LOT of learning on this one, working through some package semantics and adding some sparse lisp-unit tests. A few more days worth of effort for other chart types, and I might have something worthy of release.

I’ve been tracking my fuel efficiency for a few weeks, noting the date, gallons purchased, price per gallon, and miles since last refill. You can’t control what you don’t measure. I had a ruby script keeping up with these as YAML and then giving me some handy stats, but I’m trying to gain some levels in lisp, so I decided to fire up slime and make some graphs.

The graph isn’t pretty, but here’s my miles per gallon:

As usual, I pulled up my favorite lisp tutorial and #lisp, and after a few hours, I had a graph. Given my logging fetish, I imagine I’ll be growing the graphing capabilities in the future.

Right now, the implementation is a little weird. I have an class for a fuel entry, and then convert the date to a universal-time using net.telent.date. Then in my graphing function I use the universal time for the x values, and any number of functions for the y values (defaulting to #’mpg-of). In order to get the graph to take up most of the canvas, I then normalize the x and y values, so the minimum is 0 and the maximum is 1, and then scale the number to fit the canvas size before drawing.

I still consider myself a lisp beginner, and there are some bits I’m sure could be more efficient, but it works for now. Those interested in the code can get it here: fuelman.tar.gz, and any feedback in comments is much appreciated.

Update: Played with it a lot more, simplified my normalization crap a lot using loop instead of mapcar, and pulled it into a couple of objects: graph, graph-series, and fuel-graph-series.  Also linked the source above as a proper .tar.gz file, after reading how to do that in wordpress.  The aesthetics are still terrible, but it’s getting there: