lisp

HOWTO: start using lisp in your work environment (part 1)

Getting started with lisp is no easy task. Tools like clbuild and Lispbox make it easier than a few years ago, but there are still obstacles (some quite reasonable) to using lisp in your work environment. After conversing about the subject a little with Alberto Riva (another local lisper!) and seeing trichey mention it, I figured I’d write up the successful approach I employed.  Everyone’s workplace is different, and these may not work for you. YMMV.

First off, here are some questions any manager worth their salt will have, that you will need to address:

  1. How much is supporting another language going to cost me? (buying the software, upgrading the software, running and updating servers, training staff, etc)
  2. Will our customers be OK with using this esoteric language?
  3. How will I hire anyone who knows lisp?

In my workplace, we’re a small consulting shop, and I am both programmer and manager.  People come to us asking for a website that does X.  We then figure out they want X to solve problem P, and suggest a website Y to solve P.  These are mostly simple CRUD applications.   There’s a bunch of data and a handful of things they want done with it, but mostly they just want to look at it. We used ASP.NET with C#, and were having a hell of a time abstracting common functionality between projects.   After writing the same code over and over, we knew there had to be a better way, which brings us to the manager’s first concern: cost.

The pro-lisp argument boils down to: we can save money by adding lisp to our toolbox and using it where appropriate.

1) Establish and demonstrate inefficiencies with the current toolset

In our case, we were dealing with ASP.NET version 1, C# version 1, and .NET Framework 1.1.  This combination was vastly superior to the vbscript ASP we had been running before, but there were still things we simply couldn’t abstract.  We used code generators (provided by Microsoft and written in-house) to write tons of boilerplate C# code, and spent a lot of time frustrated.  To be fair, Microsoft tried to help by releasing more versions of ASP.NET, C#, and the .NET Framework, and it was possible to generate our C# at runtime using C#, but the syntax is ridiculous (see listing 6 on Late Binding and On-the-Fly Code Generation Using Reflection in C#).  On the flip side, all those new versions of ASP.NET, C#, and the .NET Framework require a lot of work to keep up with, and it is difficult to explain to non-technical customers why they need to spend money to get something totally invisible to them.  Another big factor in our displeasure with ASP.NET was how hard it was to have a shared user control library.

The goal of this step is to convince the manager that your current toolset is not a silver bullet, and he could be saving money by introducing other tools.  Of course, be honest.  If the current toolset is really well suited for the work, then you have no reason to switch.

2) Use open source development tools

If your workplace doesn’t already, start.  Some managers might be accustomed to paying costly fees for everything under the sun, and is rightfully skeptical about increasing his tool costs.  Some managers might be confused about the maturity and quality of open-source offerings, and applying the “you get what you pay for” adage.  Either way, the cost argument can largely disappear when using open-source lisp tools and implementations.  Ideally get emacs into your workplace.  I’d say a good 10% of my costs with running lisp working were spent learning how to use emacs effectively.  In our MS environment, we started with NAnt, Subversion, and TortoiseSVN.  Those projects convinced me that open source is inevitably going to corner the developer tools market.

The goal of this step is to convince the manager that there is no great risk to trying out new open-source projects, and OSS can be cost-saving measures.

3) Find a small project you where you can use lisp as a infrequent manual step in an existing project

By limiting the scope (particularly by excluding a runtime component), you minimize the manager’s risk if lisp isn’t up to the job, and side-step any costs with running lisp as a server, or keeping your lisp implementation updated.  In our case, we were writing a XUL application, and needed to generate tons of XML.  XUL is a neat environment, but verbose as all hell and the standard abstraction mechanisms (XSLT, XBL) left us wanting.  We introduced lisp to generate this XML, and found all the abstraction we were looking for.  Lisp was natural fit for XML generation, as the tree of code tended to mirror the resulting XML tree.  This step will take time because you’ll need to get slime/emacs/lisp configured.  This may test your manager’s patience if you’re inexperienced with lisp.

The goal of this step is to demonstrate to the manager the efficacy of lisp without requiring a major investment, and show how lisp can quickly add value to existing projects.

4) More manual lisp processes, more people

Here you apply lisp to more one-off problems, or expand on the previous project.  In our case, we started generating javascript with our XUL, wrote a pricing calculator for an existing product, and a script to check the status of many internal subdomains.  This will involve getting more people involved with lisp.  Pair program a lot here on people who don’t know lisp.  Watch SICP lectures.  In my case, my co-workers were all excited about lisp, and picked it up very easily.  We weren’t doing anything advanced, and the context of XML generation made it all pretty easy to think about.  If your co-workers do not take to lisp, then you’re probably SOL.

The goal of this step is to demonstrate to the manager that the whole team can benefit from lisp, and get them thinking about what else lisp can offer.

5) Pick a medium-sized internal project and run lisp in production environment, in cooperation with an existing project

This is where the manager needs to stick their neck out a bit and convince his powers-that-be to try something new and different.  This might involve working with your systems department to setup a new server, and will likely have pretty high visibility even if the project itself is low priority.  This project should be something used internally so we don’t have to tackle the “what would our clients think of lisp” question yet.  Here is where you really need to deliver for your manager.  In our case, we had a 10 year old C++ app running some core invoicing logic.  The thing was riddled with bugs, and the source wasn’t even in version control.  The staff had just figured out how to work around it.  This was ripe for the pickings, and we made a website that talked to the same database, so they could be used in tandem.  In our case, just about anything was better than the C++, and lisp brought a new era of flexibility.  To ease deployment, we generated lisp executables (using sb-ext:save-lisp-and-die) on the dev machines, copied them to the production server, and run them behind proxy servers.  This eliminates the costs associated with updating  lisp implementations on production servers.

The goal of this step is demonstrate lisp’s stability and effectiveness in a production environment.  If you’ve achieved this, you have demonstrated the cost-effectiveness of lisp, and can use it on any internal projects.

Gosh, that ended up a bit more verbose than I intended, I’ll try to make the next one more concise.

lisp

Comments (16)

Permalink

simple job scheduling with a threaded lisp

Yesterday I had a need to do some batch processing, making many HTTP calls to crawl a government website to pull down some public data.  I didn’t want to run this during normal hours, because I didn’t want to put a strain on their server.  I had the web crawling sorted out as a function using drakma and cl-ppcre to get the data I wanted.

I wasn’t sure how best to schedule this function to run later that night, and was about to dig into anacron and figure out the right command-line flags to sbcl, when Nathan suggested the obvious:  why not have lisp schedule it?  It was very easy to throw something together using sleep and get-universal-time.  This morning I abstracted that into a helper function using SBCL Threading:

(defun run-later (utime-to-run function)
  (sb-thread:make-thread
   #'(lambda ()
       (sleep (- utime-to-run (get-universal-time)))
       (funcall function))))

Very handy, doesn’t lock my repl, scratches my itch, and easy to write.

code snippet
lisp

Comments (1)

Permalink

logo for wedding favors

I’m getting married in a few weeks, and one of the final tasks was to make some favors for the guests. Heather and I decided on coffee mugs with a cute logo. We quickly came up with the concept of a heart with gears inside, as we’re having a steampunk/victorian themed wedding.  After one very frustrating evening with the gimp (which ended in a crash before I saved the file), I decided attempt two would be procedural, so I fired up my trusty REPL and vecto:

final version

final version

I seriously considered how to calculate the gear size, teeth, and rotation so they mesh perfectly, then gave up and specified them manually via guess-and-check.  The code is at http://paste.lisp.org/display/71465.

I was pretty happy, it ended up being one evening of hacking, with much of that taken rendering test images to get gear placement, rotation, and teeth correct.  I was able to get everything drawing in terms of one variable *r*, so I could scale the size of everything simply.  In retrospect, that could probably have been done just as easily using vecto:scale.  I used vecto:with-graphics-state more than I have before, and that made the drawing operations very straightforward.

I got a little optimistic with the gear-internal generic method, I thought to make many different styles for the inside of the gear, but ended up with only solid and spoked varieties, so had only one specialization for the generic method.

The internal shape of the gear (the number of spokes, or if it’s solid) was determined randomly, and I wrote a quick make-samples function to generate a batch of images.  Then I reviewed those with the fiancee to find a random arrangement we liked.

To pick a suitable font I found all the .ttf files on my computer, then looped through those to make a test logo for each font (the test-font function).  This proved pointless, as Heather knows her fonts and picked one from memory while I was scrolling through hundreds of png files.

The logo is off at the mug printers, and I should be getting a test mug in the mail this week.  Hopefully it will work out and I can offer secret alien technology to all my wedding guests.

code snippet
lisp
vecto

Comments (5)

Permalink

some simple cl-smtp examples

The docs on cl-smtp are a little, um, terse, so I figured I’d post a few snippets for google to find:

Sending html email with cl-smtp:

(cl-smtp:send-email
 +mail-server+
 "from-email@example.com"
 "to-email@example.com"
 "Subject"   
 "<html><body>
<p>
Shiny <strong>h</strong><em>t</em><small>m</small>l.
</p>
<p>
</body></html>"
 :extra-headers '(("Content-type" "text/html; charset=\"iso-8859-1\"")))

Sending attachments with cl-smtp:

(cl-smtp:send-email
 +mail-server+
 "from-email@example.com"
 "to-email@example.com"
 "Subject"   
 "see attachement"
 :attachments '("/path/to/attachment"))

Ok, back to my regularly scheduled slog.

code snippet
lisp

Comments (8)

Permalink

adw-charting now has at least one other user

I’ve gotten some nice email from Erik Winkels giving feedback on adw-charting, and now today (which has had already it’s share of exe-related posts), I get one more email with a patch for supporting clisp executables.

I was loading a font file using a path relative to the (asdf:component-pathname …), and that was tripping up the exe.

Building standalone EXEs is still pretty low on my list, as I mostly make web apps and have servers available.  I think the growing desire good EXE support is a sign of lisp getting some more traction, as folks get past the “wanting to get their feet wet” stage and into the “wanting other people to use their software” stage.

Erik was kind enough to send a screenshot along with this amusing note:

I’ve attached a screenshot of the utility (it’s a trading tool for EVE-Online) and I’ve deleted the working title since it can be somewhat offense for some people.

The mind reels.  What was this name that was deemed too offensive?  It looks pretty nice, maybe he’s using ltk?

screenshot of Erik's app

screenshot of Erik's app

adw-charting
lisp

Comments (1)

Permalink

adw-charting progress and plans

It’s been awhile since I posted any updates on adw-charting, but some progress has been made.  After deciding to use google charts for rendering, a few months ago, I’m re-thinking the decision.  Google charts look very nice, but they weren’t as automatic as I’d hoped, and there was still a lot of processing needed before it could display a reasonable chart either.  Using google is certainly less CPU/memory intensive than using vecto, but there are a few problems:

  • charts are limited to 300,000 pixels, which is smaller than it sounds
  • because all the data needs to be passed on a URL, it chokes on large datasets.  This can be helped by using alternate encodings for values, but that looks like a pain in the ass (basically have to map the arbitrary lisp data evenly onto 0-64 or 0-4096, then encode as ASCII, I predict rounding troubles)
  • it doesn’t automatically scale to fit your size.  For example, a bar chart uses 20px wide bars by default.  If you have too many bars they just wander off the right of the canvas, and you need to manually adjust the bar width to get things to fit
  • some of the labels / titles get cut off

The google backend certainly has it’s uses, but I’m going to revive the vecto backend, which gives complete control.  As a start, I’ve split the project into a 3 asdf systems:

  • adw-charting: has some utils and common code, declares the :adw-charting package
  • adw-charting-vecto: adds/exports functions for making vecto charts in the :adw-charting package
  • adw-charting-google: adds/exports functions for making google chart URLs into in the :adw-charting package

In this setup, you’ll load adw-charting-google.asd or adw-charting-vecto.asd, and then all the dependancies will sort themselves out, and you don’t have anything loaded you don’t need.

I looked at asdf-system-connections but didn’t see how it would help, so copied cl-xmpp‘s multiple asd file scheme.  This is currently just in the darcs repo, and may have some bugs.

I’ve also started an actual adw-charting todo list, as part of a greater effort to be more organized in my life, comments welcome here.

adw-charting
lisp
vecto

Comments (0)

Permalink

lispvan presentation

In a couple of hours I’ll be giving a presentation to lispvan about web applications with UCW and lisp.  I’ve mostly been a user of UCW, and spent time over the past week reading through the internals so I can give better explanations.  It’s a bit easier on me because UCW has been forked and modified by so many people that anything too in-depth would be inaccurate, so I have a good excuse for staying high level.

I’ve got a very small real-world example to go through, with everything running on my little eeePC, and source for a more complicated example that’s running back at home.

For reference, here’s the chart of a UCW request / response I’ll be going over:

It’s been awhile since I did any public speaking, and I’m excited about getting to sit down with other lispers.

lisp

Comments (2)

Permalink

New term: Stealthy-patch

By now we’ve all heard of monkey-patching, and I propose a new variant: stealthy-patching.  This is the act of patching a currently running process with zero downtime.  This is similar to monkey patching, but doesn’t have the negative connotations.

I just connected to the Gainesville-Green app and evaluated a few forms to add intelligent handling of the enter button on search pages, along with some new parenscript for more IE compatibility.  All told I redefined 1 UCW component and 2 defmethods, all without any active users getting interrupted.

On the other side of the coin, I also compiled and uploaded a new SBCL core containing those fixes, so next time the application restarts (in a suprise reboot, for example), my patches will still be in place.

Ahhh… so much flexibility.  It’s like getting into a hot-tub after a long day of frigid C#.

gainesville-green
lisp

Comments (0)

Permalink

Prepping for Load

A week ago, our UCW-based website (Gainesville-Green) got some local news coverage (see “Gainesville-Green.com Segment Aired on WCJB TV20” on the project’s blog), and we spent the day trying to prep for an increased load.

Our setup is fairly straightforward:

  1. apache and mod_proxy to serve static content and get dynamic content from…
  2. a lisp (sbcl) http server listening on 127.0.0.1:3xxx talking via CL-SQL to…
  3. a postgresql database on a separate, more powerful server

When 5pm rolled around, we had:

  • long-lived caches of some expensive queries in lisp
  • lisp generating proper Cache-Control and Last-Modified headers to let our mostly static content be cached aggressively (using a new UCW dispatcher)
  • some visual / usability tweaks
  • cron job to automatically restart the lisp application server if it fails to respond

Over the next couple of days we added some more:

  • used mod_disk_cache on Apache’s end to put a cache server directly in front of our lisp server
  • reworked some entry points to work nicer with the cache
  • referenced our javascript libraries from Google’s AJAX Libraries API, which will serve the big js files off their content distribution network, gzipped and minified

So, not too much optimization was done inside lisp itself besides some basic memoization of database queries.  Hans Hübner made a post on Building Resilient Web Servers last week after our sprint, but our needs were much simpler.   We have basically static content that we want built on-demand.  From what I’ve read we could probably be faster by replacing Apache with nginx or some such, but for now it meets our needs nicely.

For a few days our cron job was reporting the site as regularly down and restarting it, and we finally tracked that down to a bug in the restarting script.  Turns out the HTTP “Not Modified” response code is 304, and we were looking for something in the 2xx range, so restarting needlessly.

The burst of traffic that Friday night was the most we’ve seen on any of our lisp projects, we got over 4000 hits that day on a weaker test server.  We’re serving 500-1000 pages per day now, and most of the bandwidth is coming from google’s CDN, so our load is nice and light, averaging around 15K per request.

I LOVE how straightforward and composable all these tools are.  Now that our “stay-alive” cron job isn’t randomly killing our lisp, we should have a nice long-lived process and better uptime.

gainesville-green
lisp

Comments (4)

Permalink

lisp geocoding, more library possibilities

I’m finally working on a project that has a public component (besides a login screen), and the lisp has been flowing. I’ve been able to put in probably around 10 hours a week on this thing, and I can’t wait until its more presentable to show off. As part of this, we’re building up our work lisp codebase, and I’ve been keeping my eye out for opportunities to open source different library components, and trying to avoid creating a big ball of mud.

The first real candidate so far was adw-charting, but I think I found the next potentials: adw-yahoo and adw-google. They will be libraries for interfacing with various Yahoo! and Google services. So far I’ve only got code for one service apiece, but there’s plenty of growth potential.

Here’s a sample:

(defun geocode-fight ()
  (let ((adw-yahoo:*api-key* "YAHOO APPID")
	(adw-google:*api-key* "GOOGLE API-KEY")
	(address "5308 SW 75th ter, Gainesville FL, 32608"))
    (list
     (adw-yahoo:latlong address :cache-p nil)
     (adw-google:latlong address :cache-p nil))))

(geocode-fight)
=> (("address" 29.604265 -82.42349 "5308 SW 75th Ter")
(8 29.604923 -82.42338 "5308 SW 75th Terrace, Gainesville, FL 32608, USA"))

Yahoo and Google give back different some different data, but so far Yahoo’s coordinates seem more accurate. One thing I noticed was that Google’s geocoding service gives a different lat/long than what maps.google.com displays, which I thought was a little sneaky. I’ve also got some code to help construct google map widgets, using the homegrown html template system and parenscript, but I think much of that will end up getting stripped out, as it depends on the in-house libraries that are likely completely useless to anyone but us.

Still lots to do before either of those libs would be ready for a cl-net request, but it’s on the plate. After building so much on top of so many great free libraries, the need to contribute something back is very strong.

code snippet
lisp

Comments (0)

Permalink