Customer doesn't believe abrupt power outage damaged equipment your thoughts 10

[Rhinorhino]

Hi everyone. I have a customer with an Avaya IP office in a small hotel with 100 rooms analog handsets version 9.1.

I got to call at 5:00 in the morning on Saturday that the entire phone system was offline.

I showed up in the first thing I noticed was that they had changed television vendors from Cox to DirecTV and that the DirecTV vendor put a whole bunch of new rack mounted equipment in and plugged directly into the Avayas APC power supply backup UPS.

I'm assuming that there was an overload in an already overloaded Network closet.

When I turned the AVAYA back on after pulling their equipment off of the backup system I ended up with a couple of blown power supplies for the expansion modules and trouble with the PC that houses the voicemail pro.

I've had problems like this before but in this case the customer absolutely refuses to believe that simply unplugging or turning off a computer or phone system will cause damage like this. she's telling me this is absolutely not possible even though it happened.

I'm curious how you all will handle this or what you would say to the customer.

Is there any documentation from Avaya about abruptly shutting down phone systems.

The vendor at DirecTV who of course is not taking any responsibility is telling her that it's not their fault even though they caused the overload.

Thanks.

 

[SE_GA_IPO]

Is the IPO still in warranty period, is that why customer is skeptical? Are they trying to get out of paying for the repair?

 

[Traveler74]

Hopefully you took pictures of the closet before you did anything. If not then it may get more complicated. Sounds like DTV knew they messed up and tried to pawn it off on you guys by convincing the customer they did nothing wrong. Hopefully its under warranty and it covers situations like this if no it will get ugly. At this point I would not engage the customer any further until you brief your leadership on what's going on and get their guidance as this issue might be above your paygrade and might require the companies legal team to get involved with what it sounds like what the customer is trying to pull.

 

[gbiddlecombe]

We had several IP offices that redlighted if, when plugging in the system, the cord connected power, then jiggled enough to lose power then regained power as the plug was inserted in the outlet.

We learned to use the switch on the ups connected power strip. To turn on the power

 

[john3voltas]

When I turned the AVAYA back on after pulling their equipment off of the backup system I ended up with a couple of blown power supplies for the expansion modules and trouble with the PC that houses the voicemail pro.

Well, it seems to me we're talking about a couple of IP500 power supplies and a problem with the computer that houses the VMPro.
A couple of blown PSU's will cost peanuts.
OTOH you don't specify what kind of problem you're having with the VMPro server.
From what you're saying, if the system is a R9.1, that means it is dated. It was GA'd in late 2013 and was maintained until late 2017. Even if they bought in 2017, it is a 8 years old system. That's just for starters. No one can expect a system to work flawlessly for 8 years and not experience some issue, especially when it comes to power supply units.

I'm guessing those PSU's were already faulty for some time but never had to power-cycle because the UPS provided power at all times. That's happened to me on a couple of occasions.

Now, I'm guessing that if you replace de PSU's the IP500 and the expansion modules will start up just fine. Right? Can you confirm if you have already done that already?

Coming around the server issue, are you telling me that this adult person (your customer) has never had issues from restarting a computer without properly shutting it down?
Why does she think you've put a UPS in the first place? Have you told her the UPS was there to cope with minor power outages and more specifically to allow for proper system shutdown in case of long power outages?
It is a computer. It has a hard drive in it. That hard drive has a filesystem. That filesystem will very become corrupt over time whether it is FAT/NTFS/EXT3/EXT4. It doesn't matter. It's a filesystem that's had open files probably for years. Very easy to go corrupt and having to undergo maintenance.
It can also be a hdd malfunction. A 9yo harddrive will easily go bananas and decide to show all it's problems exactly when you need it to...boot up.

If she's a normal adult human being, she will understand all this. But I'm betting that she's not a normal adult human being, otherwise you wouldn't be here commenting with us.
Again, the couple of PSU's are a cheap replacement and you might have some in your workshop from old retired systems.
As for the server, only you can tell us what's wrong with it. If it's a simple filesystem corruption, just go ahead and reinstall the server and restore your backups. You do have backups, don't you?
If it's an hdd problem, you might have to ask her to chip in 80€ or something to replace the drive. It's not the end of the world.

Keep us up to date.
Cheers

 

[FractalTelecom]

To add to the comments already, do this:

Add up the amperage draw on every piece of equipment connected to the UPS, and the amperage pull on all the phone equipment.

Now, what is the amperage capacity of the UPS?


I always tell providers to get their own ups, or we provide a second one for them.

 

[IPOriface]

When I turned the AVAYA back on after pulling their equipment off

So the IPO & expansion modules were powered off when you arrived, you disconnected the DTV gear from the UPS, then how did you turn the Avaya gear back on? Unplug and re-plug to UPS?

Was there any damage to the UPS? I've hit dead UPS sockets more often than I've lost PSU's. A damaged UPS would be additional proof that your gear was impacted by external factors.

customer absolutely refuses to believe that simply unplugging or turning off a computer or phone system will cause damage

It shouldn't, but depending on how long that gear's been turned on for, powering it off, especially for an extended period can damage a PSU due to thermal contraction. I've not hit this a lot recently, but it was a big issue with the IP500v1 PSU's; much less so with the v2's. I have also lost expansion module PSU's here & there. A lot cheaper to replace one of them than replace the module lol.

trouble with the PC that houses the voicemail pro.

What sort of trouble? PSU dead? Won't POST? Data corruption?
If it's a dead PSU, then all signs point to a power surge than made it past the UPS and damaged 3 different pieces of equipment.

 

[Rhinorhino]

Thanks for the replies. I feel that it's coming down to the customer just does not want to look bad in front of her boss that she decided to switch to DTV and then this happened and is causing her to pay for repairs and her boss has to see the extra bill.

I feel she's trying to pretend it's faulty Avaya equipment and she should not have to pay.

DTV also keeps telling her impossible that by them plugging all their new devices into an existing vendors installation would cause an overload.

I decided not to meet the DTV tech onsite because it would just end up in a finger pointing match in front of the customer.

Yes I took lots of pictures before and after and posted notices not to plug anything into the installation.

It's the principle of the matter that it comes down to.

 

[ipohead]

As I understand it a UPS can do a proper system shutdown on a server but not an IP Office.

The office is prone to power cuts but never really an issue with the IP Office apart from the old V1 where the power supply blew. Possibly because the power went off, on, off, on in a few seconds but of course V1s had a known issue with PSUs. Replaced with and V2 PSU and this was eventually replaced with a complete V2.

 

[Rhinorhino]

How would you all handle this issue with your customer. What would you tell them.

 

[john3voltas]

@Rhinorhino
We are all still blind here. We don't know exactly what was the outcome because you're not telling us.
Is the system back up working?
Was it just a matter of replacing that couple of IP500 PSU's?
What about the server? Is it working? What was the issue with it? Was it just software (corrupted filesystem) or was there any actual hardware damage? If there was hw damage, was it the HDD unit or was it also a PSU, etc?

Handling it with the customer is easy. You're just making it seem hard. There is no system that will indefinitely work faultlessly. Eight years (give or take) is an excellent record. On top of that, it wasn't as if you were doing anything wrong. THEY did something wrong, by messing with the UPS and triggering an overload.
I don't think it ever comes any easier to explain than in this case: they did it. You fixed it. They pay.
They don't like to pay? Of course they don't, but I don't like to pay the electric bill either and yet I pay it every month.

You have three ways to deal with it.
a)
You explain. They pay. You may loose a customer but you keep your integrity. You did everything you could do.
b)
You explain. They don't pay. You keep a toxic non-paying customer and you loose your integrity.
c)
You explain. They don't pay. You legally sue. You will surely loose a customer and you will surely keep you integrity but they will learn a lesson.

You choose.
Stay cool.

Cheers

 

[Rhinorhino]

Replaced 3 module ps, finally got the PC dive with vmp running again. Removed all non IPO power connections so IPO power is now isolated. Entire system of now back online after several hours of work.

 

[biv343]

Expansion module power supplies seem to be fragile things. I had around a dozen fail at various sites a couple years back. Nothing else died, just those power bricks. I consider those consumables.

In the instances where I would provide and install a UPS, it would be labelled 'phone equipment only' and unused outlets on the UPS would have those plastic childproof outlet things inserted. A picture would be taken of the equipment room, including the UPS and saved in the customer documentation folders. If anything got plugged into that, we didn't cover it anymore. It got a lot more difficult when plugging into customer provided UPS equipment in data centers, but those customers have a vested interest in not dropping a server rack, so we very rarely had issues there.

Small doctors offices always seemed to be the worst about not wanting to pay.

 

[john3voltas]

Small doctors offices always seemed to be the worst about not wanting to pay.

I guess you're talking about MD's? I didn't have much contact with MD's offices except for dental clinics. Those were the worst to pay...and lawyers' offices too.