Well, believe it or not, there are specialized cryptography classes out there. I would never have believed it until I saw it on a course. However, the course was limited to a teenager's age so I'm not sure if it's available anymore.
But there are definitely books on it out there, like The Code Book by Simon Singh, I believe. That was the "textbook" for class.