Most of the problems I've seen with OFO have to do with one of four things.
1. Improper placement of the "snapshot" on the volume(s)
2. Not enough HD space (in the vol) to hold the "snapshot"
3. OFO minimum/maximum parameters not configured correctly
4. Not enough idle time to create the "snapshot"
By default, BE places the snapshot on the root of the volume that is being backed up. That is fine as long as there is enough free space on the volume in question (10% free space is needed, I think).
If there isn't enough space on the volume to hold the snapshot, then you must override BE and manually assign a location for it to be created by unchecking the appropriate box and typing in the full path to the location of your folder dedicated to OFO.
If you use this feature, you must remember that the snapshot probably won't be able to "grow" properly.
In the wizard, once you decide where the OFO static volume (snapshot) will reside, you can set the minimum and maximum size for the volume to grow.
If you manually assign the location, you should set both of these fields to the same value (a high one). As long as the volume never fills up to that amount you'll be OK. If it does, just adjust the sizes up until you find a setting that works. The important thing is to remember that both numbers must be the same.
If you let BE assign the location itself (on the root of the volume), then you can use the following formula to "estimate" the needed sizes:
(Num of FILES backed up) X 0.005 = (Minimum OFO Volume size needed)MB
and
(Free RAM)MB / 0.04 = (Max OFO Volume size capable in)MB
That takes care of the first three issues. The last one is a caryover from BE 7.x. By default, BE waits for a continuous block of "idle time" before creating a snapshot. Out of the box this item is set for 5 seconds. I've found that this number must be set down to 2 seconds in order to work. In BE8.5 and 8.6 it will give you a nasty "are you sure" message after you change this setting. Just press <OK> and ignore it.
In older versions of BE, you must tweak a registry key.
The path is something like:
HKEY_LOCAL_MACHINE\SOFTWARE\VERITAS\Open File Option\Config
Edit the "MinQuietTime" value and change it from 5 to 2.
The path might be a little different depending on what version you're running, but I believe the "MinQuietTime" value is named the same regardless of version.
One last thing, remember to stop and restart the BE services for any of these changes to take effect.
Good luck and keep us posted.