L grade lenses are the best optically, though quite a few non-L lenses are as good, or very nearly so. The other thing, that quite possibly accounts for much of the extra cost, is the build quality. They're more robust, built for every day professional use, and many have weather sealing, circular aperture blades etc. They hold their value well, too.
Whether they're worth the extra or not is up to you. I have L lenses, but I was also very happy with EF-S lenses when I had a 40D outfit. There are no L-grade EF-S lenses, and nothing to match to spec of the 10-22, 17-55, 15-85 - all excellent.