Ryan’s Tech Blog

I’ve recently come to an obvious conclusion: chart layout is really hard, and not in a fun way.

Luckily, Google has clarified the terms of use on their charting service, which is a RESTful API that returns png files based on some arc-ish querystring parameters. No google API key is required, the 300,000 pixel image size limit is reasonable, and they request to be notified if you need to make more than 250,000 charts in one day.

This afternoon I spent some a little time with DRAKMA, and ported my pie chart feature. I challenge you to guess which went through my layout code, and which came through google:

Give up? Me too.

Here are the two functions used to generate those charts:

(defun pie-gchart ()
  (with-gchart (:pie 300 150)
    (add-slice "foo" 10d0)
    (add-slice "bar" 10d0)
    (add-slice "baz" 20d0)
    (save-file "pie-gchart.png")))

(defun pie-chart ()
  (with-pie-chart (300 150)
    (add-slice "foo" 10d0)
    (add-slice "bar" 10d0)
    (add-slice "baz" 20d0)
    (save-file "pie-chart.png")))

I was able to use generic methods to reuse all my existing code, but instead of performing VECTO operations, it builds up the proper parameter list and performs one http call.

I plan to make all of ADW-CHARTING use google to do the hard work. I’ve got some code in the darcs repo, and after I get my other chart types ported over I’ll update the website, examples, docs, etc.

I did decide to change my API approach for the google charts, making one with-gchart that takes the chart type as an argument, instead of have a with-pie-gchart, with-line-gchart, etc.

I was a little concerned about adding the dependency on google, but all of my usages so far are either for web apps that are already connected, or for generating static html, in which I’d be saving the chart locally anyway.

Some day I might revisit the chart layout problem, but for now the parameter building problem is a lot less daunting, and the API design problem is a lot more interesting. There are tons of options for google charts, and making lispy ways to specify all that should be fun.

I apparently can’t follow instructions, and put my darcs repo in the wrong place on common-lisp.net, polluting their darcsweb. The new location is:

http://common-lisp.net/project/adw-charting/darcs/adw-charting

I’ll be deleting the old junk in a few days.

Spent a ton of time today playing with adw-charting. I wanted to expand on the source contributor graphs from the other day, and ended up implementing some basic bar chart functionality.

I’ve read a lot of complaints about the lack of lisp libraries, so whenever I sit down to do some random task, I try to use libraries as much as I can to test the validity of those complaints. I also learn a lot by reading and experimenting with other people’s code, and slowly I can see my style getting more lispish, and less C#-ish. clbuild makes it trivial to get many libraries usable, and today I had good luck with everything just working.

I used darcs as my source of dataset, its xml changes output has a simply formatted time string. I told clbuild to install closure xml (cxml), and in fairly short order I had a 80 line program that:

1. uses the cxml klacks system to pull the information I wanted out of the file in one pass
2. uses cl-ppcre to parse the username out of the darcs author string, so authors “Ryan Davis <ryan@acc…” and “ryan@acce…” both get displayed as “ryan”.
3. uses adw-charting to chart patches per day into a png

The result was less than spectacular:

The chart looks pretty nice, but its lying horribly. My data set didn’t have explicit zeros for days without activity, so the line goes from one active day to another, telling all kinds of lies. My favorite is the 2.5 patches it says I pushed on 1/11. The other fun one is that Russ shows up in the legend, but his one patch isn’t visible at all, because there was only one data point, and you need two points to make a line.

What is really appropriate here is a bar chart. Adding explicit zeros is a waste of time and RAM (among other things), so I opted for making a bar chart type. After many more hours of coding, I changed with-line-chart to with-bar-chart and re-ran my program:

Much better. There’s a few rendering problems, but the data is shown alright.

Adding support for bar charts was actually really easy, it was going back and reworking my amatuerish code that took most of the time. My big accomplishments:

• deleted a lot of code by using with-accessors
• made all usage of x/y points go through 3 functions: make-point, x, and y, allowing for easily changing the internal representation of points in the future
• removed some slots from an object, replacing it with a few functions to calculate the desired value
• added a few more generic functions to support bar charts
• reduced the size of the legend

After working on the line chart code for awhile, adding bar charts was trivial: one class that extends line-chart, one macro to mirror with-line-chart, one method to draw bars instead of lines, and a couple of helper functions. I was really pleased with how easy it was to add the different graphing mode.

Of course, giving another dataset exposes more problems:

That’s for clbuild, and there are a few issues apparent here:

• when more than one bar is shown for one day, it draws them next to each other, which makes it hard to tell where on the bar is on the x-axis
• the legend wanders off the edge
• the month/day labels aren’t very descriptive

I think I’m going to solve the first two issues by allowing a summarize mode, where less significant series can be aggregated into a single “other” series. This is how I see a lot of other programs solve it, so I’ll give that a shot. I don’t think that will completely solve the first issue, but should alleviate it a little bit.

The last issue I solved by making a function to show patches by month instead of by day:

That shows the side-by-side bars problem pretty clearly, but what I find most interesting is the right side of the chart where we see more people starting to contribute. I think that’s indicative of the project gaining more users and activity.

Another interesting one, arnesi:

Early on, Marco Baringer was the driving force, but somewhere around the middle of 2006 I guess he had finished scratching all his itches, and Atilla Lendvai became the prime mover.

Alright, I could seriously speculate about these things all night, so I’m gonna stop now.

I still need to fix the above-mentioned bar chart issues, so there’s no new release of adw-charting, but all the code mentioned here is in the public darcs repository (http://common-lisp.net/project/adw-charting/darcs/) for the curious.

See the examples folder for the darc-changes graphing program, you need to get the xml from darcs using: darcs changes –xml.

I took a break from the day-to-day work (maybe giving Visual Studio’s a timeout will solve it’s “Generation of designer file failed: Unknown server tag…” problem), and noticed jsnell posted a list of SBCL contributors to #lisp, and decided to do some graphing:

Of course, since I’ve never graphed this before, I found 2 bugs. I copied the text from jsnell paste and used cl-ppcre to split it up into data sets. I had to do a little math to get the months to line up nicely with the yearly sums, but it was all pretty straightforward.

Code, excluding the copied text from lisppaste:

(defun make-months ()
  (loop for month in '("Jan" "Feb" "Mar" "Apr" "May" "Jun"
		       "Jul" "Aug" "Sep" "Oct" "Nov" "Dec")
	counting T into val
	collect (list month (float (/ val 12)))))

(defun yearly-data ()
  (let (result)
	(cl-ppcre:do-register-groups (year contribs)
	    ("(\\d+):(\\d+)" +yearly-raw-data+)
	  ;;add 1 so the yearly totals line up with
	  ;;december monthly data
	  (push (list (+ 1 (parse-integer year))
		      (parse-integer contribs))
		result))
	result))

(defun monthly-data ()
  (let ((months (make-months))
	result)
	(cl-ppcre:do-register-groups (year monthname contribs)
	    ("(\\d+)-(\\w{3}):(\\d+)" +monthly-raw-data+)
	  (push (list
		 (+ (parse-integer year)
		    (second
		     (assoc monthname months
			    :test #'string=)))
		 (parse-integer contribs))
		result))
	result))

(defun yearly-graph ()
  (with-line-chart (600 400)
    (add-series "Yearly Contributors" (yearly-data))
    (add-series "Monthly Contributors" (monthly-data))
    ;;so the yearly totals line up, the data is offset by 1
    (set-axis :x nil :draw-gridlines-p nil :data-interval 1
	      :label-formatter #'(lambda (y)
				   (princ-to-string (1- y))))
    (set-axis :y nil)
    (save-file "yearly.png")))

Made another release tonight, but there are no new features, this is just tying up some loose ends:

• added a LICENSE file and license headers in each file.
• removed cl-vector-chart.lisp, which I had looked at for inspiration, but ultimately decided not to use.
• added a lisp file and shell script to automate making a release.

Thanks to Nikodemus Siivola for taking a look at the code and making some suggestions.

I’m still pretty new to the culture of open source, and it always amazes me when people spontaneously offer advice and help.  This sort of behavior helps restore my faith in humanity, which my national government seems to sap at every opportunity.  It’s very nice to see some upward spirals.

I’m also greatly amused by the stark contrast between the awesome lisp community that’s taking time to help me and the elitist-asshole lisp community I’ve read about off reddit.

Next time: some graphs to map performance differences as I evolve the adw-charting internals, and more Lisp.

I finally got some time to employ the adw-charting library in my intranet app, and the increase in data sets revealed a lot of bugs.  Tonight I finally put together a release.

Some changes:

• Pie chart slices are made much more intelligently, resolving some bad rendering with some datasets.
• x-axis label locations are decided in a much more sane manner, and was reworked sans scotch.
• if the x-axis label intervals are not sane, then you can specify what the interval should be with the new set-axis keyword argument, :data-interval.
• the docs are updated to mention :data-interval and show an example usage.
• Fixed a bug reported by Stuart Mackey with minimum x-values above 0.

So now I’m getting into several new territories:

1. running an open source project
2. coding lisp for performance

The first is pretty fun, really, everyone I’ve interacted with has been very helpful and reasonable.  There’s been a flurry of activity around lisp charting libraries, so I feel some sense of competition, but the big difference is I can incorporate someone else’s code to make my library better, which is pretty cool.  I’ve heard claims that open source is a way applying the scientific method to software, and (given the right licenses) that’s not too far off.  My next goal is to cannibalize cl-pdf’s charting.  They have some niceness there that I might be able to toss in front of vecto and get a lot of nice features pretty quickly.

The second is a quite a bit more daunting.  Currently I cons an absurd amount to generate the example graphs (around 70MB for one graph), and I’d like to get that down a bit.  I’m going to setup some benchmarks and track some data on time spent and bytes consed, so I can tell if what I change is helpful.  I’ve been reading up on vectors and arrays, and will try to use those instead of lists where it seems easy.    The other thing I was thinking of was putting in some type declarations as a sacrifice to the compiler gods.

Someone on #lisp started talking about making line charts using vecto, so I went ahead and got setup a common-lisp.net project: http://common-lisp.net/project/adw-charting/

Now I’m somewhat flooded with digital paperwork:

• Get my darcs repository onto common-lisp.net
• Update the documentation to point to common-lisp.net (removed all the damn lies, so only vague untruths remain)
• Get access to my trac sorted out (I sent an email to the cl.net admin)
• Get my ssh keys sorted out
• Get my setup at work pulling from the common-lisp.net darcs repo
• Get my setup at home pulling from the common-lisp.net darcs repo
• Make a script to generate docs/examples and transfer to common-lisp.net
• Make a script to generate / sign a new release
• figure out what I should be doing with mailing lists

I’ve never run an open source project before, I foresee a lot of learning in the coming weeks.

Paperwork aside, I did spend a lot of time last night on actual functionality. I spent a good bit of time reading through The Visual Display of Quantitative Information, to get a better sense of what goes into a good chart, and try to learn the right words for some of these things, and then worked on the code a bit:

• converted macros to use &body instead of &rest, which gives Emacs some better ideas on how to indent code
• REVERSEd the list of data elements before display, so the legends on pie/line charts are ordered sensibly
• Improved the example to not create duplicate X labels
• Pulled apart a monolithic function into several smaller ones, introduced a class to help pass data between them
• Tried to generate more human-friendly axis labels. Instead of splitting the y-axis into N even parts and drawing a label for whatever value happens to be there (which generates odd, unfriendly labels in the examples: 1.52, 3.05, 4.57), I look at the spread data, and pick a suitable interval. So, if the lowest y is -10 and the biggest is 2, then we have a total spread of 12, which is just over 10, so we should try to draw a label every 1.0 units. If the spread was 120, I’d try to draw a label every 10.0 units. I added some math to prevent overlapping labels, and then I have decent labels at human readable increments. There are still lots of problems with this scheme, so I haven’t pushed it yet. Hopefully I’ll have time tonight to work out some bugs, and add some more examples. This style of doing it would be a baby step towards another #lisp request, logarithmic axes.

I wasn’t very happy with using a helper class to pass state around, so I’ll probably try out a few different methods and see what works best.

My new laptop came in on Thursday, a tiny asus eeepc.  The default OS is a customized Xandros, a debian spin-off, so it was pretty easy to add some apt sources and get all the tools needed to try out clbuild.  So far I’m very impressed, almost everything has just worked.

I installed darcs to download clbuild, and then it told me to install a few other tools: cvs, svn, and git.  I had one error later in the process, when git tried to use curl, but after installing curl things went pretty cleanly.

I was unable to build SBCL from source using clbuild, but I think that was a sourceforge problem, not a clbuild problem.  Every other installation went very smoothly.

I was particularly impressed by the ability to load emacs and slime.  I find setting up emacs to be the most laborious part of getting a lisp development environment going, and the “clbuild slime” command gets me halfway there.  Looking at the clbuild script, the way they do it is pretty simple, but its a nice time savings to not need to look that stuff up.

By automatically downloading a ton of code, clbuild also provided me with some nice reading, and I spent the evening hanging out with some neighbors and browsing source code, and saw some neat patterns.

When I got back home, I hacked a little on adw-charting,  changing awhen and aif usages to when-let and if-let calls, which is a macro I saw in several Edi Weitz packages.  when-let/if-let are a little longer than Marco Baringer’s awhen/aif, but I like the explicit names.  awhen/aif introduce meaningful names into the lexical environment, and that has always made me a little nervous.  Whenever names are introduced as side-effects I’m reminded of the mess that is ruby’s ActiveRecord, which has so much magical run-time craziness that your .rb file can seem totally unrelated to what actually gets executed.

I also got just about done on a nice documentation page for adw-charting, similar to Vecto’s.  So similar, one might think I copy/pasted the Vecto content and changed the relevant bits.  I ended up using this as an excuse to try out cl-who, which is a pretty nice library.  I found a bunch of adw-charting bugs in the process.  A few people have asked for source in blog comments, so my goal was to put out a 0.5 release today, but there are still some details to clear up:

1. need to add some license crap to each source file (which might also be strangely similar to Vecto’s license crap)
2. need to add readme / license files
3. need to add some documentation about how adw-charting uses fonts
4. need to sort out gpg signing / asdf-install chores
5. possibly sort out apache incantations to allow read-only anonymous darcs access and submit a clbuild patch

I’ll try for Sunday.

Some more progress on the charting library today:

• cleaned up the API a little bit, opting for some nicer make-foo function instead of requiring calls to make-instance:
  • make-series label data &key color
  • make-axis label &key control-string draw-gridlines-p label-formatter
    • control-string: a format-compatible control string, and supplying it sets the label-formatter for the axis
    • label-formatter: a function of 1 argument, returns a string used as an axis label
  • make-line-chart width height &key series y-axis x-axis background
• The lightened portion of the graph is now the average of the background color and pure white
• The labels on the axes are optional, and the graph adjusts to fill as big an area as possible
• The distance between x-axis labes is calculated based on actual string-widths, so hopefully they will space themselves out reasonably under most conditions

Some examples:

This demonstrates using a control-string and a custom function for axis labels:

  (render-chart
   (make-line-chart 400 300
		    :background '(.7 .7 .7)
		    :series (list (make-series "SeriesA"
					       '((-1 -2) (0 4) (1 5) (4 6) (5 -3)))
				  (make-series "SeriesB"
					       '((-1 4) (0 -2) (1 6) (5 -2) (6 5))
					       :color '(.3 .7 .9))
				  (make-series "SeriesC"
					       '((-1 0) (0 3) (1 1) (2 5) (4 -6))))
		    :y-axis (make-axis "widgets"
				       :control-string "~$")
		    :x-axis (make-axis "time"
				       :label-formatter #'months-from-now->mm/yy))
   "line-chart-with-axis-labels.png")

This one changes the background color, removes the axis labels, and disabled grid lines for the x-axis:

(render-chart
   (make-line-chart 400 300
		    :background '(.7 .5 .7)
		    :series (list (make-series "SeriesA"
					       '((-1 -2) (0 4) (1 5) (4 6) (5 -3)))
				  (make-series "SeriesB"
					       '((-1 4) (0 -2) (1 6) (5 -2) (6 5))
					       :color '(.3 .7 .9))
				  (make-series "SeriesC"
					       '((-1 0) (0 3) (1 1) (2 5) (4 -6))))
		    :y-axis (make-axis nil
				       :control-string "~$")
		    :x-axis (make-axis nil
				       :draw-gridlines-p nil
				       :label-formatter #'months-from-now->mm/yy))
   "line-chart-with-axis-labels.png")

The internals are still messy, but I’ve gone through and was able to simplify some things here and there, here’s a few of the change’s I’ve made to make my code shorter and more readable:

• replacing (destructuring-bind (x y) (foo) (bar x y) with (apply #’bar (foo))
• adding a macrolet to concentrate some very similar behavior
• using loop to count up and down at the same time (for drawing labels up/down the y axis and left/right on the x axis)
• using (dolist (item lst) …) instead of (mapc #’(lambda(item) …) lst)
• using loop’s maximizing / minimizing features
• using multiple-value-list and nth to get specific values from some function results without compiler warnings

I’ve been trying to put my tongue on why exactly I like lisp so much more than other languages, and working through this made it pretty clear: I have much better tools for reducing accidental complexity than in other languages, and can get the code focusing on what I’m trying to do, not how my compiler needs me to do it. The next time I get a free afternoon, I’m going to make an imperative API, similar to vecto’s:

(with-line-chart (400 300 :background '(.7 .7 .7))
    (add-series "SeriesA"
		'((-1 -2) (0 4) (1 5) (4 6) (5 -3)))
    (add-series "SeriesB"
		'((-1 4) (0 -2) (1 6) (5 -2) (6 5))
		:color '(.3 .7 .9))
    (add-series "SeriesC"
		'((-1 0) (0 3) (1 1) (2 5) (4 -6)))
    (set-axis :y "widgets" :control-string "~$")
    (set-axis :x "time" :label-formatter #'months-from-now->mm/yy)
    (save-file "line-chart-with-axis-labels.png")

That’s pretty damn readable.

updated: fixed syntax error in the imperative sample, and got the imperative bit actually working.

Another productive afternoon:

This is the start of axes. The internals are getting very, very messy, but I really like the result. Here’s the code used to create this:

(defun line-chart-with-axis-labels ()
  "draws a simple line chart"
  (let* ((seriesA (make-instance 'series
				 :label "SeriesA"
					;data expressed as a list (x y) pairs
				 :data '((-1 -2) (0 4) (1 5) (4 6) (5 -3))))
	 (seriesB (make-instance 'series
				 :label "SeriesB"
				 :data '((-1 4) (0 -2) (1 6) (5 -2) (6 5))))
	 (y-axis (make-instance 'axis
				:label "widgets"
				:label-formatter #'(lambda (x-val)
						     (format nil "~$" x-val))))
	 (chart (make-instance 'line-chart
			       :width 400
			       :height 300
			       :background '(.7 .7 .7)
			       :series (list seriesA seriesB)
			       :y-axis y-axis)))
    (render-chart chart "line-chart-with-axis-labels.png")))

I predict adding X axes will be very similar to adding the Y axis, and then I’ll be looking more seriously at spending time to make good docs and releasing this back to the community. The API is getting a little dense, too, but that should be pretty easy to simplify, and I have a few ideas on how to do that.