A week ago, our UCW-based website (Gainesville-Green) got some local news coverage (see “Gainesville-Green.com Segment Aired on WCJB TV20” on the project’s blog), and we spent the day trying to prep for an increased load.
Our setup is fairly straightforward:
- apache and mod_proxy to serve static content and get dynamic content from…
- a lisp (sbcl) http server listening on 127.0.0.1:3xxx talking via CL-SQL to…
- a postgresql database on a separate, more powerful server
When 5pm rolled around, we had:
- long-lived caches of some expensive queries in lisp
- lisp generating proper Cache-Control and Last-Modified headers to let our mostly static content be cached aggressively (using a new UCW dispatcher)
- some visual / usability tweaks
- cron job to automatically restart the lisp application server if it fails to respond
Over the next couple of days we added some more:
- used mod_disk_cache on Apache’s end to put a cache server directly in front of our lisp server
- reworked some entry points to work nicer with the cache
- referenced our javascript libraries from Google’s AJAX Libraries API, which will serve the big js files off their content distribution network, gzipped and minified
So, not too much optimization was done inside lisp itself besides some basic memoization of database queries. Hans Hübner made a post on Building Resilient Web Servers last week after our sprint, but our needs were much simpler. We have basically static content that we want built on-demand. From what I’ve read we could probably be faster by replacing Apache with nginx or some such, but for now it meets our needs nicely.
For a few days our cron job was reporting the site as regularly down and restarting it, and we finally tracked that down to a bug in the restarting script. Turns out the HTTP “Not Modified” response code is 304, and we were looking for something in the 2xx range, so restarting needlessly.
The burst of traffic that Friday night was the most we’ve seen on any of our lisp projects, we got over 4000 hits that day on a weaker test server. We’re serving 500-1000 pages per day now, and most of the bandwidth is coming from google’s CDN, so our load is nice and light, averaging around 15K per request.
I LOVE how straightforward and composable all these tools are. Now that our “stay-alive” cron job isn’t randomly killing our lisp, we should have a nice long-lived process and better uptime.
David Rupp | 07-Jul-08 at 10:16 pm | Permalink
Someone’s doing Lisp … in Gainesville?! Why wasn’t I told?! :-)
Greetings from a Gator expat in Denver. I’ve been following your blog for a while now (via Planet Lisp), but I totally missed the Gainesville connection.
If you’re ever hiring, please let me know. My family is all in (and around) Gainesville and I’d love to get back there someday. Take a look at my blog for a taste of what I do. Mostly Java right now, but with a heritage of IBM mainframe assembler plus a passion for languages like Ruby, Python, Lisp, and Haskell / OCaml. I’ve never coded Lisp for money, but the century is young…
P.S.: Sorry for the commercial, but I couldn’t find an email address anywhere.
Holger Schauer | 08-Jul-08 at 2:26 am | Permalink
Could you say a little bit more about adding the Cache-Controlled and Last-Modified headers and the new UCW dispatcher?
patrick giagnocavo | 08-Jul-08 at 11:30 pm | Permalink
Interesting post! My company hosts a lot of dynamic web servers like those using OpenACS ; but I have been meaning to get into Lisp “one of these days”. Just a note about Postgresql performance: 1. use views wherever possible, as they are a) compiled b) reduce the amount of data that needs to go back and forth between client and server. 2. check the various cache size settings in the postgresql.conf file; tweaking some of these can give you very large performance boosts.
ryan | 09-Jul-08 at 10:15 am | Permalink
@David: we’re stealthy! We’re not hiring at the moment, but ideally some of this takes off and we’ll be in need of more lispers. If you’re ever in town on vacation or something, drop me line (I’ve emailed you my contact info), and we could chat.
@Holger: I’ll make another post on that. My co-worker Nathan put that together, so I’ll go over the details with him first to be sure I get it right.
@patrick: I’ll take a look at views, I didn’t realize they were performance boosts. I did spend an inordinate amount of time futzing with the cache size settings, and I’m frustrated that all the advice I could google was basically “there are no guidelines, just futz with it until you’re done!”