Ryan’s Tech Blog

I’m finally working on a project that has a public component (besides a login screen), and the lisp has been flowing. I’ve been able to put in probably around 10 hours a week on this thing, and I can’t wait until its more presentable to show off. As part of this, we’re building up our work lisp codebase, and I’ve been keeping my eye out for opportunities to open source different library components, and trying to avoid creating a big ball of mud.

The first real candidate so far was adw-charting, but I think I found the next potentials: adw-yahoo and adw-google. They will be libraries for interfacing with various Yahoo! and Google services. So far I’ve only got code for one service apiece, but there’s plenty of growth potential.

Here’s a sample:

(defun geocode-fight ()
  (let ((adw-yahoo:*api-key* "YAHOO APPID")
	(adw-google:*api-key* "GOOGLE API-KEY")
	(address "5308 SW 75th ter, Gainesville FL, 32608"))
    (list
     (adw-yahoo:latlong address :cache-p nil)
     (adw-google:latlong address :cache-p nil))))

(geocode-fight)
=> (("address" 29.604265 -82.42349 "5308 SW 75th Ter")
(8 29.604923 -82.42338 "5308 SW 75th Terrace, Gainesville, FL 32608, USA"))

Yahoo and Google give back different some different data, but so far Yahoo’s coordinates seem more accurate. One thing I noticed was that Google’s geocoding service gives a different lat/long than what maps.google.com displays, which I thought was a little sneaky. I’ve also got some code to help construct google map widgets, using the homegrown html template system and parenscript, but I think much of that will end up getting stripped out, as it depends on the in-house libraries that are likely completely useless to anyone but us.

Still lots to do before either of those libs would be ready for a cl-net request, but it’s on the plate. After building so much on top of so many great free libraries, the need to contribute something back is very strong.

I’ve recently come to an obvious conclusion: chart layout is really hard, and not in a fun way.

Luckily, Google has clarified the terms of use on their charting service, which is a RESTful API that returns png files based on some arc-ish querystring parameters. No google API key is required, the 300,000 pixel image size limit is reasonable, and they request to be notified if you need to make more than 250,000 charts in one day.

This afternoon I spent some a little time with DRAKMA, and ported my pie chart feature. I challenge you to guess which went through my layout code, and which came through google:

Give up? Me too.

Here are the two functions used to generate those charts:

(defun pie-gchart ()
  (with-gchart (:pie 300 150)
    (add-slice "foo" 10d0)
    (add-slice "bar" 10d0)
    (add-slice "baz" 20d0)
    (save-file "pie-gchart.png")))

(defun pie-chart ()
  (with-pie-chart (300 150)
    (add-slice "foo" 10d0)
    (add-slice "bar" 10d0)
    (add-slice "baz" 20d0)
    (save-file "pie-chart.png")))

I was able to use generic methods to reuse all my existing code, but instead of performing VECTO operations, it builds up the proper parameter list and performs one http call.

I plan to make all of ADW-CHARTING use google to do the hard work. I’ve got some code in the darcs repo, and after I get my other chart types ported over I’ll update the website, examples, docs, etc.

I did decide to change my API approach for the google charts, making one with-gchart that takes the chart type as an argument, instead of have a with-pie-gchart, with-line-gchart, etc.

I was a little concerned about adding the dependency on google, but all of my usages so far are either for web apps that are already connected, or for generating static html, in which I’d be saving the chart locally anyway.

Some day I might revisit the chart layout problem, but for now the parameter building problem is a lot less daunting, and the API design problem is a lot more interesting. There are tons of options for google charts, and making lispy ways to specify all that should be fun.

For the last few years I’ve been learning and using Lisp more, and here is a disorganized, poorly-worded dump of how Lisp changed my opinion about other languages.

Static Types (as implemented by C# and Java) are oppressive.

I spend most of my time in C#, and it always feels like I’m toiling in the Type mines under the iron fist of compiler. I frequently run into problems where I can’t abstract common code because the .NET framework doesn’t use C#’s abstraction mechanisms enough (more detail). In Lisp it feels like types are a tool I can employ when I need it, but it’s rare that I need COERCE something from one type to another. Too much of my C# is devoted to casting to / from types, a hefty tax I pay to the compiler.

Syntax doesn’t have to be so hard.

I was recently working on a ruby script, and had to stop and think: “what does ruby want for ifs again? curly braces? indentation?” In Lisp it’s easy: it wants matching parentheses. You can add syntax if you find it useful (CL-INTERPOL is one of my common additions), but there’s nothing I need to remember, no need for cheat sheets. The simplicity also highlight the strangeness of things like python’s “pass” statement. There’s no human meaning to a pass statement, it’s just to handhold the parser, and I don’t think that should be one of my responsibilities.

Development tools don’t have to be such a pain in the ass.

In C#, I frequently end up waiting for Visual Studio, and that’s on a modern dual-core workstation with 2GB of RAM. In Lisp, I end up waiting once when I start slime/emacs, and once when I initially load a big system, and then everything else is pretty instant. That’s on my Asus EEEpc, a 900MHz Celeron with 512MB RAM. I could use cores to eliminate most of that start-up time. At work we have one fast server we all use to run our lisps, and if we didn’t have to run programs like Visual Studio, we could use low-power, low-cost workstations and be perfectly happy. I know some people think Emacs is a heavyweight program, but devenv.exe sets a new standard.

Variables don’t have to be so hard.

LET statements and lexical scoping in lisp are pretty basic, and meet all my needs. The scoping rules in other languages seem really overcomplicated by comparison. In C#, sometimes curly braces open a new scope, but sometimes not. Sometimes you have to declare variables away from their usage in order for them to be available in all the right scopes you want. In python / ruby, I’m frequently confused whether I’m declaring a new variable or using one from a higher scope. Most of that is from lack of ruby/python practice, but even when first learning Lisp, the rule was so simple that I never had trouble with it.

Zach Beane asked what people are working on, here’s my contribution:

Work:

• Web application managing all data for my employer, Acceleration.net
  • Replaces an old C++ desktop app and an old ASP web app
  • Tracks LOTS of random customer data
  • Generates CSV exports for bulk credit card charging
  • Uses CLSQL and a byzantine set of libraries to use the legacy SQL Server 2000 database as the data store, allowing parallel operation with the old C++. I think this might make us the only outfit in the world using the Linux /Apache / SQL Server / Lisp stack.
  • Uses a modified UCW as a web framework, running in httpd mode behind Apache, which handles the LDAP authentication and SSL
  • Uses ADW-CHARTING for some data display
  • Uses CL-PDF and CL-TYPESETTING to generate PDF invoices, which are printed and mailed (soon to be emailed)
  • Uses many more helper libraries: CL-PPCRE, ARNESI, METABANG-BIND, CXML, PARENSCRIPT
• Code-generation for administrative web interfaces for one client (I haven’t asked for permission to disclose this, so I’m not going to mention who)
  • Builds XML files containing XUL and javascript, which get used as the front-end in an ASP.NET website
  • Uses several helper libraries: CL-PPCRE, CXML, PARENSCRIPT
• Some upcoming projects
  • various websites for profiling carbon emissions (will link when there’s anything up)
  • plan to use Postgresql with POSTMODERN, possible CL-PEREC or ELEPHANT

Personal:

• Charting random data using ADW-CHARTING, my pet graphing library
• Some experiments with chat-bots and google calendar integration to get IM updates for events. I failed at this early last year when CL-XMPP didn’t work and I didn’t know enough to read it, let alone fix it. I think cl-xmpp has seen some loving in the last year, so I might give this another shot.

I apparently can’t follow instructions, and put my darcs repo in the wrong place on common-lisp.net, polluting their darcsweb. The new location is:

http://common-lisp.net/project/adw-charting/darcs/adw-charting

I’ll be deleting the old junk in a few days.

Spent a ton of time today playing with adw-charting. I wanted to expand on the source contributor graphs from the other day, and ended up implementing some basic bar chart functionality.

I’ve read a lot of complaints about the lack of lisp libraries, so whenever I sit down to do some random task, I try to use libraries as much as I can to test the validity of those complaints. I also learn a lot by reading and experimenting with other people’s code, and slowly I can see my style getting more lispish, and less C#-ish. clbuild makes it trivial to get many libraries usable, and today I had good luck with everything just working.

I used darcs as my source of dataset, its xml changes output has a simply formatted time string. I told clbuild to install closure xml (cxml), and in fairly short order I had a 80 line program that:

1. uses the cxml klacks system to pull the information I wanted out of the file in one pass
2. uses cl-ppcre to parse the username out of the darcs author string, so authors “Ryan Davis <ryan@acc…” and “ryan@acce…” both get displayed as “ryan”.
3. uses adw-charting to chart patches per day into a png

The result was less than spectacular:

The chart looks pretty nice, but its lying horribly. My data set didn’t have explicit zeros for days without activity, so the line goes from one active day to another, telling all kinds of lies. My favorite is the 2.5 patches it says I pushed on 1/11. The other fun one is that Russ shows up in the legend, but his one patch isn’t visible at all, because there was only one data point, and you need two points to make a line.

What is really appropriate here is a bar chart. Adding explicit zeros is a waste of time and RAM (among other things), so I opted for making a bar chart type. After many more hours of coding, I changed with-line-chart to with-bar-chart and re-ran my program:

Much better. There’s a few rendering problems, but the data is shown alright.

Adding support for bar charts was actually really easy, it was going back and reworking my amatuerish code that took most of the time. My big accomplishments:

• deleted a lot of code by using with-accessors
• made all usage of x/y points go through 3 functions: make-point, x, and y, allowing for easily changing the internal representation of points in the future
• removed some slots from an object, replacing it with a few functions to calculate the desired value
• added a few more generic functions to support bar charts
• reduced the size of the legend

After working on the line chart code for awhile, adding bar charts was trivial: one class that extends line-chart, one macro to mirror with-line-chart, one method to draw bars instead of lines, and a couple of helper functions. I was really pleased with how easy it was to add the different graphing mode.

Of course, giving another dataset exposes more problems:

That’s for clbuild, and there are a few issues apparent here:

• when more than one bar is shown for one day, it draws them next to each other, which makes it hard to tell where on the bar is on the x-axis
• the legend wanders off the edge
• the month/day labels aren’t very descriptive

I think I’m going to solve the first two issues by allowing a summarize mode, where less significant series can be aggregated into a single “other” series. This is how I see a lot of other programs solve it, so I’ll give that a shot. I don’t think that will completely solve the first issue, but should alleviate it a little bit.

The last issue I solved by making a function to show patches by month instead of by day:

That shows the side-by-side bars problem pretty clearly, but what I find most interesting is the right side of the chart where we see more people starting to contribute. I think that’s indicative of the project gaining more users and activity.

Another interesting one, arnesi:

Early on, Marco Baringer was the driving force, but somewhere around the middle of 2006 I guess he had finished scratching all his itches, and Atilla Lendvai became the prime mover.

Alright, I could seriously speculate about these things all night, so I’m gonna stop now.

I still need to fix the above-mentioned bar chart issues, so there’s no new release of adw-charting, but all the code mentioned here is in the public darcs repository (http://common-lisp.net/project/adw-charting/darcs/) for the curious.

See the examples folder for the darc-changes graphing program, you need to get the xml from darcs using: darcs changes –xml.

Today and yesterday I got my eeepc setup for lisp development, using clbuild to get all the dependencies resolved for me. After some trial and error, here were my steps:

1. Added xandros repositories and updates.xepc.org repositories to /etc/apt/sources.list, set updates.xepc.org to have the same priority as the asus repository in /etc/apt/preferences
2. The version of git in those repos is too old, so also add backports.org to sources.list:
   deb http://www.backports.org/debian etch-backports main contrib non-free, then pin git-core in /etc/apt/preferences:

   Package: git-core
   
   Pin: release a=etch-backports
   
   Pin-Priority: 999

3. Install tools clbuild needs: sudo apt-get install darcs cvs subversion curl cogito git-core sbcl
4. Install clbuild: darcs get http://common-lisp.net/project/clbuild/clbuild
5. Make the script executable: chmod u+x clbuild/clbuild
6. Get a recent version of sbcl: ./clbuild/clbuild buildsbcl
7. Watch sbcl compiler output, speculate on what things like “(DEFINE-SOURCE-TRANSFORM LOGNOR ...)” might mean
8. Give up on waiting for the sbcl to finish compiling, spend 6 hours at Nathan’s house for dinner, beer, and Rock Band.
9. Get some basic libraries: ./clbuild/clbuild update --main-projects
10. Compile slime: ./clbuild/clbuild build slime
11. Load up emacs and slime to verify everything went smoothly: ./clbuild/clbuild slime

UPDATE: added bits about getting  a recent version of git installed, clbuild needs it for updating source directories after initial installation.  My repos installed 1.4.x, and that doesn’t seem to include git-config, nor set the remote.origin.url config value which clbuild depends on.  Once I got git 1.5.x, I downloaded the projects again, and now everything works fine. That took me approximately forever to figure out.

I took a break from the day-to-day work (maybe giving Visual Studio’s a timeout will solve it’s “Generation of designer file failed: Unknown server tag…” problem), and noticed jsnell posted a list of SBCL contributors to #lisp, and decided to do some graphing:

Of course, since I’ve never graphed this before, I found 2 bugs. I copied the text from jsnell paste and used cl-ppcre to split it up into data sets. I had to do a little math to get the months to line up nicely with the yearly sums, but it was all pretty straightforward.

Code, excluding the copied text from lisppaste:

(defun make-months ()
  (loop for month in '("Jan" "Feb" "Mar" "Apr" "May" "Jun"
		       "Jul" "Aug" "Sep" "Oct" "Nov" "Dec")
	counting T into val
	collect (list month (float (/ val 12)))))

(defun yearly-data ()
  (let (result)
	(cl-ppcre:do-register-groups (year contribs)
	    ("(\\d+):(\\d+)" +yearly-raw-data+)
	  ;;add 1 so the yearly totals line up with
	  ;;december monthly data
	  (push (list (+ 1 (parse-integer year))
		      (parse-integer contribs))
		result))
	result))

(defun monthly-data ()
  (let ((months (make-months))
	result)
	(cl-ppcre:do-register-groups (year monthname contribs)
	    ("(\\d+)-(\\w{3}):(\\d+)" +monthly-raw-data+)
	  (push (list
		 (+ (parse-integer year)
		    (second
		     (assoc monthname months
			    :test #'string=)))
		 (parse-integer contribs))
		result))
	result))

(defun yearly-graph ()
  (with-line-chart (600 400)
    (add-series "Yearly Contributors" (yearly-data))
    (add-series "Monthly Contributors" (monthly-data))
    ;;so the yearly totals line up, the data is offset by 1
    (set-axis :x nil :draw-gridlines-p nil :data-interval 1
	      :label-formatter #'(lambda (y)
				   (princ-to-string (1- y))))
    (set-axis :y nil)
    (save-file "yearly.png")))

Made another release tonight, but there are no new features, this is just tying up some loose ends:

• added a LICENSE file and license headers in each file.
• removed cl-vector-chart.lisp, which I had looked at for inspiration, but ultimately decided not to use.
• added a lisp file and shell script to automate making a release.

Thanks to Nikodemus Siivola for taking a look at the code and making some suggestions.

I’m still pretty new to the culture of open source, and it always amazes me when people spontaneously offer advice and help.  This sort of behavior helps restore my faith in humanity, which my national government seems to sap at every opportunity.  It’s very nice to see some upward spirals.

I’m also greatly amused by the stark contrast between the awesome lisp community that’s taking time to help me and the elitist-asshole lisp community I’ve read about off reddit.

Next time: some graphs to map performance differences as I evolve the adw-charting internals, and more Lisp.

I finally got some time to employ the adw-charting library in my intranet app, and the increase in data sets revealed a lot of bugs.  Tonight I finally put together a release.

Some changes:

• Pie chart slices are made much more intelligently, resolving some bad rendering with some datasets.
• x-axis label locations are decided in a much more sane manner, and was reworked sans scotch.
• if the x-axis label intervals are not sane, then you can specify what the interval should be with the new set-axis keyword argument, :data-interval.
• the docs are updated to mention :data-interval and show an example usage.
• Fixed a bug reported by Stuart Mackey with minimum x-values above 0.

So now I’m getting into several new territories:

1. running an open source project
2. coding lisp for performance

The first is pretty fun, really, everyone I’ve interacted with has been very helpful and reasonable.  There’s been a flurry of activity around lisp charting libraries, so I feel some sense of competition, but the big difference is I can incorporate someone else’s code to make my library better, which is pretty cool.  I’ve heard claims that open source is a way applying the scientific method to software, and (given the right licenses) that’s not too far off.  My next goal is to cannibalize cl-pdf’s charting.  They have some niceness there that I might be able to toss in front of vecto and get a lot of nice features pretty quickly.

The second is a quite a bit more daunting.  Currently I cons an absurd amount to generate the example graphs (around 70MB for one graph), and I’d like to get that down a bit.  I’m going to setup some benchmarks and track some data on time spent and bytes consed, so I can tell if what I change is helpful.  I’ve been reading up on vectors and arrays, and will try to use those instead of lists where it seems easy.    The other thing I was thinking of was putting in some type declarations as a sacrifice to the compiler gods.