A week ago, our UCW-based website (Gainesville-Green) got some local news coverage (see “Gainesville-Green.com Segment Aired on WCJB TV20” on the project’s blog), and we spent the day trying to prep for an increased load.
Our setup is fairly straightforward:
- apache and mod_proxy to serve static content and get dynamic content from…
- a lisp (sbcl) http server listening on 127.0.0.1:3xxx talking via CL-SQL to…
- a postgresql database on a separate, more powerful server
When 5pm rolled around, we had:
- long-lived caches of some expensive queries in lisp
- lisp generating proper Cache-Control and Last-Modified headers to let our mostly static content be cached aggressively (using a new UCW dispatcher)
- some visual / usability tweaks
- cron job to automatically restart the lisp application server if it fails to respond
Over the next couple of days we added some more:
- used mod_disk_cache on Apache’s end to put a cache server directly in front of our lisp server
- reworked some entry points to work nicer with the cache
- referenced our javascript libraries from Google’s AJAX Libraries API, which will serve the big js files off their content distribution network, gzipped and minified
So, not too much optimization was done inside lisp itself besides some basic memoization of database queries. Hans Hübner made a post on Building Resilient Web Servers last week after our sprint, but our needs were much simpler.  We have basically static content that we want built on-demand. From what I’ve read we could probably be faster by replacing Apache with nginx or some such, but for now it meets our needs nicely.
For a few days our cron job was reporting the site as regularly down and restarting it, and we finally tracked that down to a bug in the restarting script. Turns out the HTTP “Not Modified” response code is 304, and we were looking for something in the 2xx range, so restarting needlessly.
The burst of traffic that Friday night was the most we’ve seen on any of our lisp projects, we got over 4000 hits that day on a weaker test server. We’re serving 500-1000 pages per day now, and most of the bandwidth is coming from google’s CDN, so our load is nice and light, averaging around 15K per request.
I LOVE how straightforward and composable all these tools are. Now that our “stay-alive” cron job isn’t randomly killing our lisp, we should have a nice long-lived process and better uptime.
4 Comments