Ryan’s Tech Blog

Spent a ton of time today playing with adw-charting. I wanted to expand on the source contributor graphs from the other day, and ended up implementing some basic bar chart functionality.

I’ve read a lot of complaints about the lack of lisp libraries, so whenever I sit down to do some random task, I try to use libraries as much as I can to test the validity of those complaints. I also learn a lot by reading and experimenting with other people’s code, and slowly I can see my style getting more lispish, and less C#-ish. clbuild makes it trivial to get many libraries usable, and today I had good luck with everything just working.

I used darcs as my source of dataset, its xml changes output has a simply formatted time string. I told clbuild to install closure xml (cxml), and in fairly short order I had a 80 line program that:

1. uses the cxml klacks system to pull the information I wanted out of the file in one pass
2. uses cl-ppcre to parse the username out of the darcs author string, so authors “Ryan Davis <ryan@acc…” and “ryan@acce…” both get displayed as “ryan”.
3. uses adw-charting to chart patches per day into a png

The result was less than spectacular:

The chart looks pretty nice, but its lying horribly. My data set didn’t have explicit zeros for days without activity, so the line goes from one active day to another, telling all kinds of lies. My favorite is the 2.5 patches it says I pushed on 1/11. The other fun one is that Russ shows up in the legend, but his one patch isn’t visible at all, because there was only one data point, and you need two points to make a line.

What is really appropriate here is a bar chart. Adding explicit zeros is a waste of time and RAM (among other things), so I opted for making a bar chart type. After many more hours of coding, I changed with-line-chart to with-bar-chart and re-ran my program:

Much better. There’s a few rendering problems, but the data is shown alright.

Adding support for bar charts was actually really easy, it was going back and reworking my amatuerish code that took most of the time. My big accomplishments:

• deleted a lot of code by using with-accessors
• made all usage of x/y points go through 3 functions: make-point, x, and y, allowing for easily changing the internal representation of points in the future
• removed some slots from an object, replacing it with a few functions to calculate the desired value
• added a few more generic functions to support bar charts
• reduced the size of the legend

After working on the line chart code for awhile, adding bar charts was trivial: one class that extends line-chart, one macro to mirror with-line-chart, one method to draw bars instead of lines, and a couple of helper functions. I was really pleased with how easy it was to add the different graphing mode.

Of course, giving another dataset exposes more problems:

That’s for clbuild, and there are a few issues apparent here:

• when more than one bar is shown for one day, it draws them next to each other, which makes it hard to tell where on the bar is on the x-axis
• the legend wanders off the edge
• the month/day labels aren’t very descriptive

I think I’m going to solve the first two issues by allowing a summarize mode, where less significant series can be aggregated into a single “other” series. This is how I see a lot of other programs solve it, so I’ll give that a shot. I don’t think that will completely solve the first issue, but should alleviate it a little bit.

The last issue I solved by making a function to show patches by month instead of by day:

That shows the side-by-side bars problem pretty clearly, but what I find most interesting is the right side of the chart where we see more people starting to contribute. I think that’s indicative of the project gaining more users and activity.

Another interesting one, arnesi:

Early on, Marco Baringer was the driving force, but somewhere around the middle of 2006 I guess he had finished scratching all his itches, and Atilla Lendvai became the prime mover.

Alright, I could seriously speculate about these things all night, so I’m gonna stop now.

I still need to fix the above-mentioned bar chart issues, so there’s no new release of adw-charting, but all the code mentioned here is in the public darcs repository (http://common-lisp.net/project/adw-charting/darcs/) for the curious.

See the examples folder for the darc-changes graphing program, you need to get the xml from darcs using: darcs changes –xml.

Made another release tonight, but there are no new features, this is just tying up some loose ends:

• added a LICENSE file and license headers in each file.
• removed cl-vector-chart.lisp, which I had looked at for inspiration, but ultimately decided not to use.
• added a lisp file and shell script to automate making a release.

Thanks to Nikodemus Siivola for taking a look at the code and making some suggestions.

I’m still pretty new to the culture of open source, and it always amazes me when people spontaneously offer advice and help.  This sort of behavior helps restore my faith in humanity, which my national government seems to sap at every opportunity.  It’s very nice to see some upward spirals.

I’m also greatly amused by the stark contrast between the awesome lisp community that’s taking time to help me and the elitist-asshole lisp community I’ve read about off reddit.

Next time: some graphs to map performance differences as I evolve the adw-charting internals, and more Lisp.

Someone on #lisp started talking about making line charts using vecto, so I went ahead and got setup a common-lisp.net project: http://common-lisp.net/project/adw-charting/

Now I’m somewhat flooded with digital paperwork:

• Get my darcs repository onto common-lisp.net
• Update the documentation to point to common-lisp.net (removed all the damn lies, so only vague untruths remain)
• Get access to my trac sorted out (I sent an email to the cl.net admin)
• Get my ssh keys sorted out
• Get my setup at work pulling from the common-lisp.net darcs repo
• Get my setup at home pulling from the common-lisp.net darcs repo
• Make a script to generate docs/examples and transfer to common-lisp.net
• Make a script to generate / sign a new release
• figure out what I should be doing with mailing lists

I’ve never run an open source project before, I foresee a lot of learning in the coming weeks.

Paperwork aside, I did spend a lot of time last night on actual functionality. I spent a good bit of time reading through The Visual Display of Quantitative Information, to get a better sense of what goes into a good chart, and try to learn the right words for some of these things, and then worked on the code a bit:

• converted macros to use &body instead of &rest, which gives Emacs some better ideas on how to indent code
• REVERSEd the list of data elements before display, so the legends on pie/line charts are ordered sensibly
• Improved the example to not create duplicate X labels
• Pulled apart a monolithic function into several smaller ones, introduced a class to help pass data between them
• Tried to generate more human-friendly axis labels. Instead of splitting the y-axis into N even parts and drawing a label for whatever value happens to be there (which generates odd, unfriendly labels in the examples: 1.52, 3.05, 4.57), I look at the spread data, and pick a suitable interval. So, if the lowest y is -10 and the biggest is 2, then we have a total spread of 12, which is just over 10, so we should try to draw a label every 1.0 units. If the spread was 120, I’d try to draw a label every 10.0 units. I added some math to prevent overlapping labels, and then I have decent labels at human readable increments. There are still lots of problems with this scheme, so I haven’t pushed it yet. Hopefully I’ll have time tonight to work out some bugs, and add some more examples. This style of doing it would be a baby step towards another #lisp request, logarithmic axes.

I wasn’t very happy with using a helper class to pass state around, so I’ll probably try out a few different methods and see what works best.

I use quicken to manage my finances, and find it overall does an excellent job, but every now and then it does something outrageous:

Seriously?   You still can’t handle that?  I guess that means its time to take another stab at gnucash.

There are several tutorials out there on how to do this, but they all differed a little, and figured I might as well note mine.  I’m replicating the setup I have at work onto a home computer, so this is my second time around.

1. Download and install Eclipse, the java version should be just fine
2. Pick a directory for your flash projects.  For this guide, I’m using the Eclipse default, “C:\Documents and Settings\Ryan\workspace”
3. Download the Flex 2 SDK, unzip it into a “Flex 2 SDK” folder in your project directory
4. Download the Flex Ant tasks, and unzip the file into a directory in your project directory
5. Install whatever version control tools you prefer
6. Open Eclipse, get to the workspace view
7. Make a new “General” project for the Flex 2 SDK
8. Go to Window->Preferences, select the Ant->Runtime node in the tree
9. Go to the “Classpath” tab, select “Ant Home Entries (default)”, and then click “Add External JARs”
10. Select the flexTasks.jar from the flex ant folder.
11. Click OK
12. Go to the “Tasks” tab, select “Add Task”
13. Name it “mxmlc”, and choose the flexTasks.jar from the dropdown
14. In the tree view, navigate to / -> flex2 -> ant, and then select MxmlcTask.class from the right pane
15. Click OK
16. Go to the “Properties” tab, select “Add Property”
17. Name it “FLEX_HOME”, and make the path to your Flex 2 SDK folder
18. Click OK
19. Click OK

Now you’re ready to start the hard part, actually making your flash program.

Gonna try to be quick about this, as there’s chicken on the grill. I took a break from Team Fortress 2 to play some more with Vecto, another fine library from Xach. I started another post about last weekend’s lisp playing, but this one got finished first.

I’ve been tracking my fuel efficiency for a few weeks, noting the date, gallons purchased, price per gallon, and miles since last refill. You can’t control what you don’t measure. I had a ruby script keeping up with these as YAML and then giving me some handy stats, but I’m trying to gain some levels in lisp, so I decided to fire up slime and make some graphs.

The graph isn’t pretty, but here’s my miles per gallon:

As usual, I pulled up my favorite lisp tutorial and #lisp, and after a few hours, I had a graph. Given my logging fetish, I imagine I’ll be growing the graphing capabilities in the future.

Right now, the implementation is a little weird. I have an class for a fuel entry, and then convert the date to a universal-time using net.telent.date. Then in my graphing function I use the universal time for the x values, and any number of functions for the y values (defaulting to #’mpg-of). In order to get the graph to take up most of the canvas, I then normalize the x and y values, so the minimum is 0 and the maximum is 1, and then scale the number to fit the canvas size before drawing.

I still consider myself a lisp beginner, and there are some bits I’m sure could be more efficient, but it works for now. Those interested in the code can get it here: fuelman.tar.gz, and any feedback in comments is much appreciated.

Update: Played with it a lot more, simplified my normalization crap a lot using loop instead of mapcar, and pulled it into a couple of objects: graph, graph-series, and fuel-graph-series.  Also linked the source above as a proper .tar.gz file, after reading how to do that in wordpress.  The aesthetics are still terrible, but it’s getting there:

Mucking about with Shibboleth again, and ran into some errors on the SP, specifically:

2007-07-10 19:49:42 DEBUG SAML.libcurl [79] sessionGet: SSL read: error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate, errno 0

This is the “catch-all” error message for Shibboleth OpenSSL Errors. After much research and testing, it was a problem with the IdP, their server was rejecting my client certificate.

How it should work (I think)

1. User requests protected content on the SP
2. SP redirects to the IdP for authentication
3. IdP authenticates user, sends an SSL (with client certificae) SOAP request to the SP with some info, and then redirects the user back
4. SP validates SOAP request by comparing client certifacte with a white-list in the shibboleth metadata
5. SP sends an SSL (with client certificate) SOAP request to IdP to get more information about the user (in my case, a username so I can identify them in my database)
6. IdP validates SSL cert of SP with a white-list in their shibboleth metadata, responds with whatever information was requested
7. SP uses that information to serve or deny access to the user from step 1

SSL cert problems can happen at steps 4 and 6. You can test these somewhat by using openssl on the command line. Openssl has about a brazillion options, but the one useful here is s_client(1).

Be sure the SP recognizes the IdP’s certs

First up, check that the SP has the IdP’s certs in order:

openssl s_client -connect HOST:443 -showcerts

That will give you back the certificate chain:

Certificate chain
0 s: SUBJECT FOR THE IDP
i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)05/CN=VeriSign Class 3 Secure Server CA
—–BEGIN CERTIFICATE—–
ASCII JUMBLE
—–END CERTIFICATE—–
1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)05/CN=VeriSign Class 3 Secure Server CA
i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
—–BEGIN CERTIFICATE—–
ASCII JUMBLE
—–END CERTIFICATE—–
—
Server certificate
subject=SUBJECT FOR THE IDP
issuer=/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)05/CN=VeriSign Class 3 Secure Server CA
—
some other stuff…

The issuing certificate (in the example, cert 1) should be in your shibboleth metadata. All you need is the top-most issuer, and shibboleth will look down the certificat chain until it finds a certificate it trusts, and all will be well.

If that doesn’t solve it, then we see if the IdP has the SP cert straight.

Be sure the IdP recognizes the SP’s certs

This is harder to debug from the SP’s side, but you can try make an SSL connection using the SP cert and key:

openssl s_client -connect HOST:443 -showcerts -cert SP.crt -key SP.key

If that connects, then you know the server isn’t rejecting you outright, but it’s possible that Apache config is rejecting you elsewhere. The best way to check is to just ask your IdP if your config is still in their shibboleth white-list.

If that doesn’t solve it, check the apache config.

Be sure everyone’s apache config is OK

The possible failure here is that the certificate is being rejected before making it to shibboleth. You don’t want to tell your whole server to accept any client certificate, you want to just pass those through to shibboleth and let shibboleth decide. You’ll want to have SSLVerifyClient optional_no_ca in your Apache location blocks for shibboleth URLs.

Conclusion

Certificates are a good idea, but a pain in the ass. The shibboleth-users mailing list is a good source of information, and you can get prompt replies from there. And next time I have this problem, I’ll know where to look for debugging tips.

For a recent project I wanted to authenticate using Central Authentication Service (CAS), a single-sign on server deployed world-wide. My project is in ASP.NET, so I hunted down CASP, a C# class produced by John Tantalo at Case Western Reserve University. Coincidentally, John was also responsible for Planarity, a flash game which has only stolen mere days of my life.

I had a few nits to pick with it, so at the risk of calling his baby ugly:

1. Doesn't handle the latest CAS protocol, CAS2
2. Dumps the authenticated username into Session, which isn't what I wanted
3. Doesn't use "using" statements when dealing with IDisposable objects
4. Doesn't use XML comments, which .NET tools prefer
5. Some minor duplication in specifying the login URL and the validation URL.

So, all in all nothing really big. I ended up going a little nuts with it resolving all my complaints. It can now speak CAS1 or CAS2, and has a bunch of options that I added in to solve my specific needs. It seems a little overcomplicated now, but I always get that feeling when I'm thinking in C#.

Installation

Pretty simple:

1. Download the source: CASP.cs (BSD license)
2. Add it to your project

Example usage

Like Tantalo's CASP, mine is designed to be used from a System.Web.UI.Page, and will redirect the browser about as needed.

Simplest example, uses CAS2 by default

Slightly more complex, using CAS1 and always renewing the authentication ticket

Most complex example, giving you flexibility to decide what to do about errors, etc

This code is certified:

I recently saw a post via reddit on Rediscovering the button element about making nice buttons with little icons, and was introduced to the Silk Icons available from Mark James. Mark has released these under a Creative Commons Attribution license, any my UIs will be better for it.

I went through last night and gave a little face-lift to one of my intranet projects, and it looks a lot nicer now. I added icons to buttons and links using CSS.

Here's the css I used:

and so forth, with a *-icon for the different icons I wanted. Then, I decorated my buttons, links, and what-have-you by adding a class="icon foo-icon" attribute. There are other ways accomplish that with CSS, but I liked this scheme the best. In the end, I have added nothing significant to the markup, but made everything else look a lot nicer:

Thanks a lot, Mark!

I saw an announcement today that CodePlex, Microsoft's version of Sourceforge, has released a source control client. From the release:

A common theme we've heard from our users is the desire to be able to work offline (in the "edit-merge-commit" style) when working on their CodePlex projects. Six months ago, we started working to write such a client that would integrate with our existing TFS server infrastructure, and today we've released our first beta of the client.

The CodePlex Client is a command line client for Windows, and requires .NET 2.0.

This infuriates me. This cool thing they spent six months (six!) writing is called Subversion, and it had a 1.0.0 release three years ago. Subversion had its first beta in late 2003, so the Codeplex folks are waaay behind the state of the art on this one.

As a whole, I think the state of software is abysmal. The only way to make it better is to stop writing new code. New code is always full of bugs, and its an expensive path to get from blank screen to stable program. We need to treat programming more like math, we need to build on our results. Development tools is a special market, as our needs are all very similar, and when we need a tool, we have the skills to make it.

The Codeplex staff stated they needed to write their own client in order to integrate with the TFS server infrastructure. According to an msdn article (Get All Your Devs In A Row With Visual Studio 2005 Team System), TFS seems to be a complicated tool to help manage your developers. Reading the description, TFS seems to be an issue tracker, unit tester, continuous integration, source control system, and visual studio plugin. So, basically a combination of Trac, NUnit, CruiseControl.NET, Subversion, and a visual studio plugin. Why not just write the visual studio plugin, and hook into the tools people are already using? All those tools have rich plugin-architectures that would probably support any sensible addition you'd want to make.

This problem is ingrained at Microsoft, which feels the need to brand everything, but it is in no way limited to them. A search on Sourceforge for "issue tracker" gives 585 results. Sifting through those to pick a winner is difficult.

It's more fun to write new code than read old code, but this fun wears off. After a certain initial momentum creating your new tool, you will inevitably come to a realization "this is going to take me for-fucking-ever". Unless your itch is particularly strong, you'll probably quit, and the world will be cursed with a 586th buggy issue tracker. By writing a plugin, you can ride the new-code high usually from start to finish, since its a much smaller task.

Reading code seems more difficult, but I think that's largely perception. Its just another puzzle to solve. Once you get over the idea that reading code is more difficult, it's really not that bad. For most mature projects, it's probably easier for you to read through someone else's mound of debugged code than it is to write and debug your own mud-ball.

I think we need find and evolve extensible tools, and stop trying to write them over again. We can get the custom behavior we all need by writing and debugging plugins, which are going to be orders of magnitude faster and cheaper than writing the whole system from scratch. I see this happening already, with communities forming around different tools to share plugins.

Next time you need a development tool, don't open a new code file. Do us all a favor, open up a browser, and just re-use previous results.