Skip to content

some simple cl-smtp examples

The docs on cl-smtp are a little, um, terse, so I figured I’d post a few snippets for google to find:

Sending html email with cl-smtp:

(cl-smtp:send-email
 +mail-server+
 "from-email@example.com"
 "to-email@example.com"
 "Subject"   
 "<html><body>
<p>
Shiny <strong>h</strong><em>t</em><small>m</small>l.
</p>
<p>
</body></html>"
 :extra-headers '(("Content-type" "text/html; charset=\"iso-8859-1\"")))

Sending attachments with cl-smtp:

(cl-smtp:send-email
 +mail-server+
 "from-email@example.com"
 "to-email@example.com"
 "Subject"   
 "see attachement"
 :attachments '("/path/to/attachment"))

Ok, back to my regularly scheduled slog.

adw-charting now has at least one other user

I’ve gotten some nice email from Erik Winkels giving feedback on adw-charting, and now today (which has had already it’s share of exe-related posts), I get one more email with a patch for supporting clisp executables.

I was loading a font file using a path relative to the (asdf:component-pathname …), and that was tripping up the exe.

Building standalone EXEs is still pretty low on my list, as I mostly make web apps and have servers available.  I think the growing desire good EXE support is a sign of lisp getting some more traction, as folks get past the “wanting to get their feet wet” stage and into the “wanting other people to use their software” stage.

Erik was kind enough to send a screenshot along with this amusing note:

I’ve attached a screenshot of the utility (it’s a trading tool for EVE-Online) and I’ve deleted the working title since it can be somewhat offense for some people.

The mind reels.  What was this name that was deemed too offensive?  It looks pretty nice, maybe he’s using ltk?

screenshot of Erik's app

screenshot of Erik's app

adw-charting progress and plans

It’s been awhile since I posted any updates on adw-charting, but some progress has been made.  After deciding to use google charts for rendering, a few months ago, I’m re-thinking the decision.  Google charts look very nice, but they weren’t as automatic as I’d hoped, and there was still a lot of processing needed before it could display a reasonable chart either.  Using google is certainly less CPU/memory intensive than using vecto, but there are a few problems:

  • charts are limited to 300,000 pixels, which is smaller than it sounds
  • because all the data needs to be passed on a URL, it chokes on large datasets.  This can be helped by using alternate encodings for values, but that looks like a pain in the ass (basically have to map the arbitrary lisp data evenly onto 0-64 or 0-4096, then encode as ASCII, I predict rounding troubles)
  • it doesn’t automatically scale to fit your size.  For example, a bar chart uses 20px wide bars by default.  If you have too many bars they just wander off the right of the canvas, and you need to manually adjust the bar width to get things to fit
  • some of the labels / titles get cut off

The google backend certainly has it’s uses, but I’m going to revive the vecto backend, which gives complete control.  As a start, I’ve split the project into a 3 asdf systems:

  • adw-charting: has some utils and common code, declares the :adw-charting package
  • adw-charting-vecto: adds/exports functions for making vecto charts in the :adw-charting package
  • adw-charting-google: adds/exports functions for making google chart URLs into in the :adw-charting package

In this setup, you’ll load adw-charting-google.asd or adw-charting-vecto.asd, and then all the dependancies will sort themselves out, and you don’t have anything loaded you don’t need.

I looked at asdf-system-connections but didn’t see how it would help, so copied cl-xmpp‘s multiple asd file scheme.  This is currently just in the darcs repo, and may have some bugs.

I’ve also started an actual adw-charting todo list, as part of a greater effort to be more organized in my life, comments welcome here.

lispvan presentation

In a couple of hours I’ll be giving a presentation to lispvan about web applications with UCW and lisp.  I’ve mostly been a user of UCW, and spent time over the past week reading through the internals so I can give better explanations.  It’s a bit easier on me because UCW has been forked and modified by so many people that anything too in-depth would be inaccurate, so I have a good excuse for staying high level.

I’ve got a very small real-world example to go through, with everything running on my little eeePC, and source for a more complicated example that’s running back at home.

For reference, here’s the chart of a UCW request / response I’ll be going over:

It’s been awhile since I did any public speaking, and I’m excited about getting to sit down with other lispers.

New term: Stealthy-patch

By now we’ve all heard of monkey-patching, and I propose a new variant: stealthy-patching.  This is the act of patching a currently running process with zero downtime.  This is similar to monkey patching, but doesn’t have the negative connotations.

I just connected to the Gainesville-Green app and evaluated a few forms to add intelligent handling of the enter button on search pages, along with some new parenscript for more IE compatibility.  All told I redefined 1 UCW component and 2 defmethods, all without any active users getting interrupted.

On the other side of the coin, I also compiled and uploaded a new SBCL core containing those fixes, so next time the application restarts (in a suprise reboot, for example), my patches will still be in place.

Ahhh… so much flexibility.  It’s like getting into a hot-tub after a long day of frigid C#.

Prepping for Load

A week ago, our UCW-based website (Gainesville-Green) got some local news coverage (see “Gainesville-Green.com Segment Aired on WCJB TV20” on the project’s blog), and we spent the day trying to prep for an increased load.

Our setup is fairly straightforward:

  1. apache and mod_proxy to serve static content and get dynamic content from…
  2. a lisp (sbcl) http server listening on 127.0.0.1:3xxx talking via CL-SQL to…
  3. a postgresql database on a separate, more powerful server

When 5pm rolled around, we had:

  • long-lived caches of some expensive queries in lisp
  • lisp generating proper Cache-Control and Last-Modified headers to let our mostly static content be cached aggressively (using a new UCW dispatcher)
  • some visual / usability tweaks
  • cron job to automatically restart the lisp application server if it fails to respond

Over the next couple of days we added some more:

  • used mod_disk_cache on Apache’s end to put a cache server directly in front of our lisp server
  • reworked some entry points to work nicer with the cache
  • referenced our javascript libraries from Google’s AJAX Libraries API, which will serve the big js files off their content distribution network, gzipped and minified

So, not too much optimization was done inside lisp itself besides some basic memoization of database queries.  Hans Hübner made a post on Building Resilient Web Servers last week after our sprint, but our needs were much simpler.   We have basically static content that we want built on-demand.  From what I’ve read we could probably be faster by replacing Apache with nginx or some such, but for now it meets our needs nicely.

For a few days our cron job was reporting the site as regularly down and restarting it, and we finally tracked that down to a bug in the restarting script.  Turns out the HTTP “Not Modified” response code is 304, and we were looking for something in the 2xx range, so restarting needlessly.

The burst of traffic that Friday night was the most we’ve seen on any of our lisp projects, we got over 4000 hits that day on a weaker test server.  We’re serving 500-1000 pages per day now, and most of the bandwidth is coming from google’s CDN, so our load is nice and light, averaging around 15K per request.

I LOVE how straightforward and composable all these tools are.  Now that our “stay-alive” cron job isn’t randomly killing our lisp, we should have a nice long-lived process and better uptime.

lisp geocoding, more library possibilities

I’m finally working on a project that has a public component (besides a login screen), and the lisp has been flowing. I’ve been able to put in probably around 10 hours a week on this thing, and I can’t wait until its more presentable to show off. As part of this, we’re building up our work lisp codebase, and I’ve been keeping my eye out for opportunities to open source different library components, and trying to avoid creating a big ball of mud.

The first real candidate so far was adw-charting, but I think I found the next potentials: adw-yahoo and adw-google. They will be libraries for interfacing with various Yahoo! and Google services. So far I’ve only got code for one service apiece, but there’s plenty of growth potential.

Here’s a sample:

(defun geocode-fight ()
  (let ((adw-yahoo:*api-key* "YAHOO APPID")
	(adw-google:*api-key* "GOOGLE API-KEY")
	(address "5308 SW 75th ter, Gainesville FL, 32608"))
    (list
     (adw-yahoo:latlong address :cache-p nil)
     (adw-google:latlong address :cache-p nil))))

(geocode-fight)
=> (("address" 29.604265 -82.42349 "5308 SW 75th Ter")
(8 29.604923 -82.42338 "5308 SW 75th Terrace, Gainesville, FL 32608, USA"))

Yahoo and Google give back different some different data, but so far Yahoo’s coordinates seem more accurate. One thing I noticed was that Google’s geocoding service gives a different lat/long than what maps.google.com displays, which I thought was a little sneaky. I’ve also got some code to help construct google map widgets, using the homegrown html template system and parenscript, but I think much of that will end up getting stripped out, as it depends on the in-house libraries that are likely completely useless to anyone but us.

Still lots to do before either of those libs would be ready for a cl-net request, but it’s on the plate. After building so much on top of so many great free libraries, the need to contribute something back is very strong.

adw-charting to get a lot prettier

I’ve recently come to an obvious conclusion: chart layout is really hard, and not in a fun way.

Luckily, Google has clarified the terms of use on their charting service, which is a RESTful API that returns png files based on some arc-ish querystring parameters. No google API key is required, the 300,000 pixel image size limit is reasonable, and they request to be notified if you need to make more than 250,000 charts in one day.

This afternoon I spent some a little time with DRAKMA, and ported my pie chart feature. I challenge you to guess which went through my layout code, and which came through google:

pie-gchart.png

pie-chart.png

Give up? Me too.

Here are the two functions used to generate those charts:

(defun pie-gchart ()
  (with-gchart (:pie 300 150)
    (add-slice "foo" 10d0)
    (add-slice "bar" 10d0)
    (add-slice "baz" 20d0)
    (save-file "pie-gchart.png")))

(defun pie-chart ()
  (with-pie-chart (300 150)
    (add-slice "foo" 10d0)
    (add-slice "bar" 10d0)
    (add-slice "baz" 20d0)
    (save-file "pie-chart.png")))

I was able to use generic methods to reuse all my existing code, but instead of performing VECTO operations, it builds up the proper parameter list and performs one http call.

I plan to make all of ADW-CHARTING use google to do the hard work. I’ve got some code in the darcs repo, and after I get my other chart types ported over I’ll update the website, examples, docs, etc.

I did decide to change my API approach for the google charts, making one with-gchart that takes the chart type as an argument, instead of have a with-pie-gchart, with-line-gchart, etc.

I was a little concerned about adding the dependency on google, but all of my usages so far are either for web apps that are already connected, or for generating static html, in which I’d be saving the chart locally anyway.

Some day I might revisit the chart layout problem, but for now the parameter building problem is a lot less daunting, and the API design problem is a lot more interesting. There are tons of options for google charts, and making lispy ways to specify all that should be fun.

What learning lisp taught me about other languages

For the last few years I’ve been learning and using Lisp more, and here is a disorganized, poorly-worded dump of how Lisp changed my opinion about other languages.

Static Types (as implemented by C# and Java) are oppressive.

I spend most of my time in C#, and it always feels like I’m toiling in the Type mines under the iron fist of compiler. I frequently run into problems where I can’t abstract common code because the .NET framework doesn’t use C#’s abstraction mechanisms enough (more detail). In Lisp it feels like types are a tool I can employ when I need it, but it’s rare that I need COERCE something from one type to another. Too much of my C# is devoted to casting to / from types, a hefty tax I pay to the compiler.

Syntax doesn’t have to be so hard.

I was recently working on a ruby script, and had to stop and think: “what does ruby want for ifs again? curly braces? indentation?” In Lisp it’s easy: it wants matching parentheses. You can add syntax if you find it useful (CL-INTERPOL is one of my common additions), but there’s nothing I need to remember, no need for cheat sheets. The simplicity also highlight the strangeness of things like python’s “pass” statement. There’s no human meaning to a pass statement, it’s just to handhold the parser, and I don’t think that should be one of my responsibilities.

Development tools don’t have to be such a pain in the ass.

In C#, I frequently end up waiting for Visual Studio, and that’s on a modern dual-core workstation with 2GB of RAM. In Lisp, I end up waiting once when I start slime/emacs, and once when I initially load a big system, and then everything else is pretty instant. That’s on my Asus EEEpc, a 900MHz Celeron with 512MB RAM. I could use cores to eliminate most of that start-up time. At work we have one fast server we all use to run our lisps, and if we didn’t have to run programs like Visual Studio, we could use low-power, low-cost workstations and be perfectly happy. I know some people think Emacs is a heavyweight program, but devenv.exe sets a new standard.

Variables don’t have to be so hard.

LET statements and lexical scoping in lisp are pretty basic, and meet all my needs. The scoping rules in other languages seem really overcomplicated by comparison. In C#, sometimes curly braces open a new scope, but sometimes not. Sometimes you have to declare variables away from their usage in order for them to be available in all the right scopes you want. In python / ruby, I’m frequently confused whether I’m declaring a new variable or using one from a higher scope. Most of that is from lack of ruby/python practice, but even when first learning Lisp, the rule was so simple that I never had trouble with it.

First flash game is out!

We finally pushed our first flash game, Celest, out the door at work! The project was quite a learning experience. The point of the game is to get a spaceship into orbit with simple directional thrusters.

Check it out at celest.funfinds.us:

We’ve starting posting about those learning experiences on the celest blog:

Enjoy! Look at our ads!