in 1999 i wrote a paper on security in set-top boxes (one of my first papers); yay. but, one approach we had was to build a custom class loader that would actually load encrypted classes.
the details of the paper are:
1999 - Security in Set-Top boxes European Multimedia, Embedded Systems and Electronic Commerce EMMSEC '99, Stockholm, SWEDEN June 21-23, 1999
I believe encryption is probably not worth the effort for most people, it's not much of a hurdle to overcome. Ultimately all custom class loader execution paths will lead to a ClassLoader.defineClass call which can either be intercepted by creating a modified system class loader or by creating a JVMPI agent which listens for JVMPI_EVENT_CLASS_LOAD_HOOK events. If you can prevent these methods from being employed then you're probably operating in a secure environment where encryption would be overkill.
another alternative = encrypted class files (Score:5, Interesting)
the details of the paper are:
1999 - Security in Set-Top boxes
European Multimedia, Embedded Systems and Electronic Commerce
EMMSEC '99, Stockholm, SWEDEN
June 21-23, 1999
COPY: (pdf)
http://www.ardiri.com/publications/emmsec9 9
there was a lot of interest on this topic back in the time
Re:another alternative = encrypted class files (Score:4, Informative)