This is a very difficult topic to discuss, simply because of the terminology confusion. In my experience, an instance of WAS refers to a seperate complete install of WAS on a machine, or an install on a completely different machine. This sort of thing should only really be done if you want to get involved in clustering. Remember, even if you have seperate instances (installs) of WAS on a machine, you will still have to keep the server ports different - but WAS will not be able to manage this for you.
As far as seperate JVM's, each Application Server runs in its own JVM. Each JVM can have a heap size up to 1 gig (larger in some cases). Enterprise applications running on the same Application Server will not run in seperate JVM's.
Hope this helps.