I'd also check through the backup log to determine what time each backup selection takes.
File backups should be the quickest, provided the supporting infrastructure is there, in my experience, Exchange is slower. It would help to know exactly the types of backups being performed.