Ryan’s Tech Blog

Today and yesterday I got my eeepc setup for lisp development, using clbuild to get all the dependencies resolved for me. After some trial and error, here were my steps:

1. Added xandros repositories and updates.xepc.org repositories to /etc/apt/sources.list, set updates.xepc.org to have the same priority as the asus repository in /etc/apt/preferences
2. The version of git in those repos is too old, so also add backports.org to sources.list:
   deb http://www.backports.org/debian etch-backports main contrib non-free, then pin git-core in /etc/apt/preferences:

   Package: git-core
   
   Pin: release a=etch-backports
   
   Pin-Priority: 999

3. Install tools clbuild needs: sudo apt-get install darcs cvs subversion curl cogito git-core sbcl
4. Install clbuild: darcs get http://common-lisp.net/project/clbuild/clbuild
5. Make the script executable: chmod u+x clbuild/clbuild
6. Get a recent version of sbcl: ./clbuild/clbuild buildsbcl
7. Watch sbcl compiler output, speculate on what things like “(DEFINE-SOURCE-TRANSFORM LOGNOR ...)” might mean
8. Give up on waiting for the sbcl to finish compiling, spend 6 hours at Nathan’s house for dinner, beer, and Rock Band.
9. Get some basic libraries: ./clbuild/clbuild update --main-projects
10. Compile slime: ./clbuild/clbuild build slime
11. Load up emacs and slime to verify everything went smoothly: ./clbuild/clbuild slime

UPDATE: added bits about getting  a recent version of git installed, clbuild needs it for updating source directories after initial installation.  My repos installed 1.4.x, and that doesn’t seem to include git-config, nor set the remote.origin.url config value which clbuild depends on.  Once I got git 1.5.x, I downloaded the projects again, and now everything works fine. That took me approximately forever to figure out.

I took a break from the day-to-day work (maybe giving Visual Studio’s a timeout will solve it’s “Generation of designer file failed: Unknown server tag…” problem), and noticed jsnell posted a list of SBCL contributors to #lisp, and decided to do some graphing:

Of course, since I’ve never graphed this before, I found 2 bugs. I copied the text from jsnell paste and used cl-ppcre to split it up into data sets. I had to do a little math to get the months to line up nicely with the yearly sums, but it was all pretty straightforward.

Code, excluding the copied text from lisppaste:

(defun make-months ()
  (loop for month in '("Jan" "Feb" "Mar" "Apr" "May" "Jun"
		       "Jul" "Aug" "Sep" "Oct" "Nov" "Dec")
	counting T into val
	collect (list month (float (/ val 12)))))

(defun yearly-data ()
  (let (result)
	(cl-ppcre:do-register-groups (year contribs)
	    ("(\\d+):(\\d+)" +yearly-raw-data+)
	  ;;add 1 so the yearly totals line up with
	  ;;december monthly data
	  (push (list (+ 1 (parse-integer year))
		      (parse-integer contribs))
		result))
	result))

(defun monthly-data ()
  (let ((months (make-months))
	result)
	(cl-ppcre:do-register-groups (year monthname contribs)
	    ("(\\d+)-(\\w{3}):(\\d+)" +monthly-raw-data+)
	  (push (list
		 (+ (parse-integer year)
		    (second
		     (assoc monthname months
			    :test #'string=)))
		 (parse-integer contribs))
		result))
	result))

(defun yearly-graph ()
  (with-line-chart (600 400)
    (add-series "Yearly Contributors" (yearly-data))
    (add-series "Monthly Contributors" (monthly-data))
    ;;so the yearly totals line up, the data is offset by 1
    (set-axis :x nil :draw-gridlines-p nil :data-interval 1
	      :label-formatter #'(lambda (y)
				   (princ-to-string (1- y))))
    (set-axis :y nil)
    (save-file "yearly.png")))

Made another release tonight, but there are no new features, this is just tying up some loose ends:

• added a LICENSE file and license headers in each file.
• removed cl-vector-chart.lisp, which I had looked at for inspiration, but ultimately decided not to use.
• added a lisp file and shell script to automate making a release.

Thanks to Nikodemus Siivola for taking a look at the code and making some suggestions.

I’m still pretty new to the culture of open source, and it always amazes me when people spontaneously offer advice and help.  This sort of behavior helps restore my faith in humanity, which my national government seems to sap at every opportunity.  It’s very nice to see some upward spirals.

I’m also greatly amused by the stark contrast between the awesome lisp community that’s taking time to help me and the elitist-asshole lisp community I’ve read about off reddit.

Next time: some graphs to map performance differences as I evolve the adw-charting internals, and more Lisp.

I finally got some time to employ the adw-charting library in my intranet app, and the increase in data sets revealed a lot of bugs.  Tonight I finally put together a release.

Some changes:

• Pie chart slices are made much more intelligently, resolving some bad rendering with some datasets.
• x-axis label locations are decided in a much more sane manner, and was reworked sans scotch.
• if the x-axis label intervals are not sane, then you can specify what the interval should be with the new set-axis keyword argument, :data-interval.
• the docs are updated to mention :data-interval and show an example usage.
• Fixed a bug reported by Stuart Mackey with minimum x-values above 0.

So now I’m getting into several new territories:

1. running an open source project
2. coding lisp for performance

The first is pretty fun, really, everyone I’ve interacted with has been very helpful and reasonable.  There’s been a flurry of activity around lisp charting libraries, so I feel some sense of competition, but the big difference is I can incorporate someone else’s code to make my library better, which is pretty cool.  I’ve heard claims that open source is a way applying the scientific method to software, and (given the right licenses) that’s not too far off.  My next goal is to cannibalize cl-pdf’s charting.  They have some niceness there that I might be able to toss in front of vecto and get a lot of nice features pretty quickly.

The second is a quite a bit more daunting.  Currently I cons an absurd amount to generate the example graphs (around 70MB for one graph), and I’d like to get that down a bit.  I’m going to setup some benchmarks and track some data on time spent and bytes consed, so I can tell if what I change is helpful.  I’ve been reading up on vectors and arrays, and will try to use those instead of lists where it seems easy.    The other thing I was thinking of was putting in some type declarations as a sacrifice to the compiler gods.

Someone on #lisp started talking about making line charts using vecto, so I went ahead and got setup a common-lisp.net project: http://common-lisp.net/project/adw-charting/

Now I’m somewhat flooded with digital paperwork:

• Get my darcs repository onto common-lisp.net
• Update the documentation to point to common-lisp.net (removed all the damn lies, so only vague untruths remain)
• Get access to my trac sorted out (I sent an email to the cl.net admin)
• Get my ssh keys sorted out
• Get my setup at work pulling from the common-lisp.net darcs repo
• Get my setup at home pulling from the common-lisp.net darcs repo
• Make a script to generate docs/examples and transfer to common-lisp.net
• Make a script to generate / sign a new release
• figure out what I should be doing with mailing lists

I’ve never run an open source project before, I foresee a lot of learning in the coming weeks.

Paperwork aside, I did spend a lot of time last night on actual functionality. I spent a good bit of time reading through The Visual Display of Quantitative Information, to get a better sense of what goes into a good chart, and try to learn the right words for some of these things, and then worked on the code a bit:

• converted macros to use &body instead of &rest, which gives Emacs some better ideas on how to indent code
• REVERSEd the list of data elements before display, so the legends on pie/line charts are ordered sensibly
• Improved the example to not create duplicate X labels
• Pulled apart a monolithic function into several smaller ones, introduced a class to help pass data between them
• Tried to generate more human-friendly axis labels. Instead of splitting the y-axis into N even parts and drawing a label for whatever value happens to be there (which generates odd, unfriendly labels in the examples: 1.52, 3.05, 4.57), I look at the spread data, and pick a suitable interval. So, if the lowest y is -10 and the biggest is 2, then we have a total spread of 12, which is just over 10, so we should try to draw a label every 1.0 units. If the spread was 120, I’d try to draw a label every 10.0 units. I added some math to prevent overlapping labels, and then I have decent labels at human readable increments. There are still lots of problems with this scheme, so I haven’t pushed it yet. Hopefully I’ll have time tonight to work out some bugs, and add some more examples. This style of doing it would be a baby step towards another #lisp request, logarithmic axes.

I wasn’t very happy with using a helper class to pass state around, so I’ll probably try out a few different methods and see what works best.

My new laptop came in on Thursday, a tiny asus eeepc.  The default OS is a customized Xandros, a debian spin-off, so it was pretty easy to add some apt sources and get all the tools needed to try out clbuild.  So far I’m very impressed, almost everything has just worked.

I installed darcs to download clbuild, and then it told me to install a few other tools: cvs, svn, and git.  I had one error later in the process, when git tried to use curl, but after installing curl things went pretty cleanly.

I was unable to build SBCL from source using clbuild, but I think that was a sourceforge problem, not a clbuild problem.  Every other installation went very smoothly.

I was particularly impressed by the ability to load emacs and slime.  I find setting up emacs to be the most laborious part of getting a lisp development environment going, and the “clbuild slime” command gets me halfway there.  Looking at the clbuild script, the way they do it is pretty simple, but its a nice time savings to not need to look that stuff up.

By automatically downloading a ton of code, clbuild also provided me with some nice reading, and I spent the evening hanging out with some neighbors and browsing source code, and saw some neat patterns.

When I got back home, I hacked a little on adw-charting,  changing awhen and aif usages to when-let and if-let calls, which is a macro I saw in several Edi Weitz packages.  when-let/if-let are a little longer than Marco Baringer’s awhen/aif, but I like the explicit names.  awhen/aif introduce meaningful names into the lexical environment, and that has always made me a little nervous.  Whenever names are introduced as side-effects I’m reminded of the mess that is ruby’s ActiveRecord, which has so much magical run-time craziness that your .rb file can seem totally unrelated to what actually gets executed.

I also got just about done on a nice documentation page for adw-charting, similar to Vecto’s.  So similar, one might think I copy/pasted the Vecto content and changed the relevant bits.  I ended up using this as an excuse to try out cl-who, which is a pretty nice library.  I found a bunch of adw-charting bugs in the process.  A few people have asked for source in blog comments, so my goal was to put out a 0.5 release today, but there are still some details to clear up:

1. need to add some license crap to each source file (which might also be strangely similar to Vecto’s license crap)
2. need to add readme / license files
3. need to add some documentation about how adw-charting uses fonts
4. need to sort out gpg signing / asdf-install chores
5. possibly sort out apache incantations to allow read-only anonymous darcs access and submit a clbuild patch

I’ll try for Sunday.

Some more progress on the charting library today:

• cleaned up the API a little bit, opting for some nicer make-foo function instead of requiring calls to make-instance:
  • make-series label data &key color
  • make-axis label &key control-string draw-gridlines-p label-formatter
    • control-string: a format-compatible control string, and supplying it sets the label-formatter for the axis
    • label-formatter: a function of 1 argument, returns a string used as an axis label
  • make-line-chart width height &key series y-axis x-axis background
• The lightened portion of the graph is now the average of the background color and pure white
• The labels on the axes are optional, and the graph adjusts to fill as big an area as possible
• The distance between x-axis labes is calculated based on actual string-widths, so hopefully they will space themselves out reasonably under most conditions

Some examples:

This demonstrates using a control-string and a custom function for axis labels:

  (render-chart
   (make-line-chart 400 300
		    :background '(.7 .7 .7)
		    :series (list (make-series "SeriesA"
					       '((-1 -2) (0 4) (1 5) (4 6) (5 -3)))
				  (make-series "SeriesB"
					       '((-1 4) (0 -2) (1 6) (5 -2) (6 5))
					       :color '(.3 .7 .9))
				  (make-series "SeriesC"
					       '((-1 0) (0 3) (1 1) (2 5) (4 -6))))
		    :y-axis (make-axis "widgets"
				       :control-string "~$")
		    :x-axis (make-axis "time"
				       :label-formatter #'months-from-now->mm/yy))
   "line-chart-with-axis-labels.png")

This one changes the background color, removes the axis labels, and disabled grid lines for the x-axis:

(render-chart
   (make-line-chart 400 300
		    :background '(.7 .5 .7)
		    :series (list (make-series "SeriesA"
					       '((-1 -2) (0 4) (1 5) (4 6) (5 -3)))
				  (make-series "SeriesB"
					       '((-1 4) (0 -2) (1 6) (5 -2) (6 5))
					       :color '(.3 .7 .9))
				  (make-series "SeriesC"
					       '((-1 0) (0 3) (1 1) (2 5) (4 -6))))
		    :y-axis (make-axis nil
				       :control-string "~$")
		    :x-axis (make-axis nil
				       :draw-gridlines-p nil
				       :label-formatter #'months-from-now->mm/yy))
   "line-chart-with-axis-labels.png")

The internals are still messy, but I’ve gone through and was able to simplify some things here and there, here’s a few of the change’s I’ve made to make my code shorter and more readable:

• replacing (destructuring-bind (x y) (foo) (bar x y) with (apply #’bar (foo))
• adding a macrolet to concentrate some very similar behavior
• using loop to count up and down at the same time (for drawing labels up/down the y axis and left/right on the x axis)
• using (dolist (item lst) …) instead of (mapc #’(lambda(item) …) lst)
• using loop’s maximizing / minimizing features
• using multiple-value-list and nth to get specific values from some function results without compiler warnings

I’ve been trying to put my tongue on why exactly I like lisp so much more than other languages, and working through this made it pretty clear: I have much better tools for reducing accidental complexity than in other languages, and can get the code focusing on what I’m trying to do, not how my compiler needs me to do it. The next time I get a free afternoon, I’m going to make an imperative API, similar to vecto’s:

(with-line-chart (400 300 :background '(.7 .7 .7))
    (add-series "SeriesA"
		'((-1 -2) (0 4) (1 5) (4 6) (5 -3)))
    (add-series "SeriesB"
		'((-1 4) (0 -2) (1 6) (5 -2) (6 5))
		:color '(.3 .7 .9))
    (add-series "SeriesC"
		'((-1 0) (0 3) (1 1) (2 5) (4 -6)))
    (set-axis :y "widgets" :control-string "~$")
    (set-axis :x "time" :label-formatter #'months-from-now->mm/yy)
    (save-file "line-chart-with-axis-labels.png")

That’s pretty damn readable.

updated: fixed syntax error in the imperative sample, and got the imperative bit actually working.

Another productive afternoon:

This is the start of axes. The internals are getting very, very messy, but I really like the result. Here’s the code used to create this:

(defun line-chart-with-axis-labels ()
  "draws a simple line chart"
  (let* ((seriesA (make-instance 'series
				 :label "SeriesA"
					;data expressed as a list (x y) pairs
				 :data '((-1 -2) (0 4) (1 5) (4 6) (5 -3))))
	 (seriesB (make-instance 'series
				 :label "SeriesB"
				 :data '((-1 4) (0 -2) (1 6) (5 -2) (6 5))))
	 (y-axis (make-instance 'axis
				:label "widgets"
				:label-formatter #'(lambda (x-val)
						     (format nil "~$" x-val))))
	 (chart (make-instance 'line-chart
			       :width 400
			       :height 300
			       :background '(.7 .7 .7)
			       :series (list seriesA seriesB)
			       :y-axis y-axis)))
    (render-chart chart "line-chart-with-axis-labels.png")))

I predict adding X axes will be very similar to adding the Y axis, and then I’ll be looking more seriously at spending time to make good docs and releasing this back to the community. The API is getting a little dense, too, but that should be pretty easy to simplify, and I have a few ideas on how to do that.

It now does line charts and better pie charts, and has had a lot of bugs removed since the last post. We did some code review on it at work, brought some of the spaghetti under control, and included the library in an intranet app.

Here’s a sample line graph:

The (somewhat verbose) code for this:

(defun line-chart-sample ()
  "draws a simple line chart"
  (let* ((seriesA (make-instance 'series
				 :label "SeriesA"
					;data expressed as a list (x y) pairs
				 :data '((-1 -2) (0 4) (1 5) (4 6) (5 -3))))
	 (seriesB (make-instance 'series
				 :label "SeriesB"
				 :data '((-1 4) (0 -2) (1 6) (5 -2) (6 5))))
	 (chart (make-instance 'line-chart
			       :width 400
			       :background '(.7 .7 .7)
			       :series (list seriesA seriesB))))
    (render-chart chart "line-chart-sample.png")))

I’ll probably make some easier syntax for creating series, similar to the one I made for pie slices, but for now the CLOS approach has been working decently. My data is coming from clsql, so the code ends up more concise than this example.

There are a few spots internally where it gets a little nasty, needing to pass many things around to many functions. I may end up changing the style to use “with-line-chart” functions with some global variables tracking the current chart, but I’m not sure how that would work in a multi-threaded environment. Hell, I’m not sure if my existing code will work in a multi-threaded environment. The load on this intranet app is very low, so I may be lucky enough to get away with it.

I’ve seen a lot of complaints about the lisp community being arrogant bastards who are too busy being smarter than you to help. Over the past few months, I’ve been observing behavior, lurking in #lisp, and I think I’ve figured out the source of confusion. The lisp community behaves differently than many other communities, and many lisp newbies have no way to know that.

Based on the many blog posts and articles extolling lisp, as a newcomer I expected the room to have:

• jubilant evangelists hopping at the opportunity to make another convert
• experts offering silver-bullet solutions to every problem
• rapid-fire discussion on how to solve hard problems (with silver-bullet solutions)

Obviously, I was wrong.

• There are no evangelists on call to help new lisp users. Some people have the channel open, and will offer advice, but there’s no one chomping at the bit.
• experts will offer solutions to problems, if you ask an intelligent question
• occasional rapid-fire discussion about hard problems that are WAAAY over my head
• occasional discussion about simpler problems
• lots of redirecting to web pages with information

I think the hype around languages has been taken up a notch since ruby and rails hit the scene (my brother and I would joke about the hyperbole by trading suggestions over IM like “why don’t you just $gem install my_internal_webapps”). Language mavens get practically orgasmic in their praise, so much as to spawn a mock blog: Lisp at Light Speed. That thing makes me laugh every time I read it. I’m noticing a trend: the real core members of a community are too busy working to blog, and most evangelists don’t have a lot of knowledge, they just feel really good about what they’ve done so far.

I’ve asked easy / stupid questions in #lisp without any noticeable backlash by treating it as if I were a 6th grader in the university library, surrounded by Ph.D students who are all working quietly on their own theses. This is how I handle any knowledgeable group that I’m trying to get help from. I check every reference manual I can find first (clhs, practical common lisp, google), if I have code I first copy/paste it into pastebin, then include that link along with a lengthy, detailed question. I re-read my question a few times, thinking if I could phrase it any better, then submit. For now, all my questions have been basic enough that someone will link me to the relevant resource or give a terse answer almost immediately.

I have seen some people ask questions without much forethought, and get criticized bluntly, which too frequently gets interpreted as harshly. I saw someone post some code, claiming to have solved some problem for the community, and the terse response was something like “that code is not very good”. It’s the truth, but that’s gotta sting. I tried to soften it by eliciting some specific recommendations for the newcomer, but I think that person will simply walk away thinking #lisp is full of assholes.

In my mental image, that sixth grader just walked up to the Ph.Ds with some chicken scratch on notebook paper and said “Hey, I did this for you”. The Ph.Ds, being engrossed in their work, look up and immediately think “there’s no way I could ever use that”. Being used to talking to other Ph.Ds, they dismiss the kid’s paper, trying not to be mean about it, and then go back to work.

It seems to me that much of the negative impression around the lisp community arises from sky-high expectations and bad question asking procedures.