AFAIK this doesn't work as one might expect (I ran into the same issue).
The priority you can configure for a server flow only dictates in which order they are checked to see which one matches the call. Nothing else.
The only failover options SBC provides is HA (obviously) and within the Routing configuration.
So if you have two IPOs you can have two IPs set up in your SIP Server config and add both to the same routing profile. As per the documentation SBC does indeed take availability into account here (Routing), so if Prio 1 is down it will take Prio 2:
"Before routing the SIP traffic to the available SIP servers, Avaya SBC monitors the SIP server status and uses the server statusinformation to exclude the unavailable SIP servers. To know the availableservers information and to route the SIP traffic to the availableSIP servers, Avaya SBC uses the Heartbeat feature configured on the server entity."
But that won't help you with two different B interfaces..
If you need failover from one interface to the other, then you need to setup two trunks between IPO and SBC A side and two trunks between SBC B side and the provider.
Then IPO has to handle the failover in ARS. See
https://support.avaya.com/ext/index?page=content&id=SOLN307244 where it states "As per Avaya SME, the only failover the SBC’s handle natively is in HA mode. Everything external to it handles any reroutes, either the service provider on one on side or whatever call server (CM, IPO, ASM) on the other side."