Routers are usually slowed down by anything policy based, such as QoS or access lists. If these are not necessary then remove them. The reason a router is slower than a switch is because it has to read more of the packet before it can forward it, plus it has to do routing table lookups. So by reducing the size of the routing table, amount of access-lists (and their size) and any other policy based functions will speed up your router.
On the other hand you could always upgrade and implement MLS