" Once you transfer and no one picks up it does not go to VM but back to the operator."
So you are transferring to an extension (other than the Operator), and it is not covering to the extension's voice mail?
Make sure that your extensions with mailboxes are in the voice mail coverage group, typically group 30, but it could be any group from 1 to 30.
Menu
System Programming
Start
Extensions
More
Group Coverage
30
Enter
Inspect
See if your extensions are listed. If not, back up and enter them. This assumes that it is group 30. You should at least see the Operator's extension, since that one covers to the AA.
Good luck!!