I have a fairly good understanding of cryptography.
As far as math goes, most modern cryptography requires a good understanding of some number theory (modular arithmetic, prime numbers, and finite field arithmetic), as well as a good understanding of Linear Algebra, some calculus, and boolean algebra.
It's really not that hard to figure out how algorithms work and how to code them, the trouble comes when analyzing them to determine the amount of secrecy they provide, and performing cryptanalysis. The best way to start learning Cryptography (IMHO) is to start with the classical ciphers such as Caesar, vigenere, Hill, etc..., and work your way up from there to the modern ciphers.
A really nice learning tool that I've used to explain cryptographic algorithms is Cryptool (http://www.cryptool.com/
). It allows you to cipher and decipher text on almost every cipher known to man, has some attack and analysis tools, and has some really nice interactive things built into it that show you how the cryptographic algorithms work. I highly recommend you download it (unfortunately it doesn't work in Linux very well...).
There are several good books on Cryptography. A lot of people mention Bruce Schneier books on Cryptography, but I think they lack the mathematical detail that is needed to really understand how a cipher works, and what the weaknesses are. His Applied Cryptography book doesn't even cover AES (or the rijndael cipher), and that's what everyone uses nowadays.
I would recommend finding a college level book on Cryptography if you really want to understand it.
Also there is a lot of documentation on the web. There are several FIPS (Federal Information Processing Standard) publications covering different areas of cryptography that are all good reads, and there are Wikipedia articles on just about every encryption algorithm known to man.
I hope this information helps. Good luck on your adventures into the world of Cryptography.