INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Jobs

9608 VPN issue - Invalid Configuration

9608 VPN issue - Invalid Configuration

(OP)
I have configured our 9608 phone using the below configuration
It appears to exchange keys and fails after 15 secs.

The error is VPN Tunnel Failure - Invalid Configuration

Does anyone know what this generic error actually means?

Is there a way I can access the log files of the phone to see the detailed error?

Any help would be appreciated





################################################## #
## VPN Mode
## 0: Disabled, 1: Enabled.
################################################## #

SET NVVPNMODE 1

################################################## #
## Vendor.
## 1: Juniper/Netscreen, 2. Cisco
## 3: CheckPoint/ Nokia 4: Other
## 5: Nortel.
################################################## #
SET NVVPNSVENDOR 1

################################################## #
## GATEWAY
################################################## #
SET NVSGIP **PUBLIC IP OF REMOTE ENDPOINT**

################################################## #
## Encapsulation Type.
## 0: 4500-4500, 1: Disabled
## 2: 2070-500, 3: ?
## 4: RFC (500-500)
################################################## #
SET NVVPNENCAPS 0

################################################## #
## Copy TOS.
## 1: Yes, 2: No
################################################## #
SET NVVPNCOPYTOS 2

################################################## #
## Authentication Type.
##
## [For Cisco/Juniper/Checkpoint/Other]
## 3: PSK, 4: PSK with Xauth
## 5: RSA signatures with Xauth, 6: Hybrid Xauth
## 7: RSA signatures.
##
## [Nortel Authentication Type]
## 1: Local credentials, 2: Radius Credentials.
## 3: Radius SecureID, 4: Radius Axent.
################################################## #
SET NVVPNAUTHTYPE 3

################################################## #
## Preshared KEY PSK
##################################################
SET NVIKEPSK "PSKKEY"

################################################## #
## VPN User Type.
## 1: Any, 2: User
################################################## #
SET NVVPNUSERTYPE 1

################################################## #
## VPN User name.
################################################## #
SET NVVPNUSER mscep1

################################################## #
## Password Type.
## 1: Save in Flash, 2: Erase on reset
## 3: Numeric OTP, 4: Alpha-Numeric OTP
## 5: Erase on VPN termination.
################################################## #
SET NVVPNPSWDTYPE 1

################################################## #
## User Password.
################################################## #
SET NVVPNPSWD mscep1

################################################## #
## IKE ID (Group Name).
################################################## #
SET NVIKEID mscep

################################################## #
## IKE ID Type.
## 1: IPv4_ADDR, 2: FQDN
## 3: USER_FQDN, 9: DER_ASN1_DN
## 11: Key ID
################################################## #
SET NVIKEIDTYPE 1

################################################## #
## IKE Xchg Mode.
## 1: Aggressive, 2: Identity Protect.
################################################## #
SET NVIKEXCHGMODE 2

################################################## #
## IKE DH Group.
################################################## #
SET NVIKEDHGRP 14

################################################## #
## IKE Encryption Algo.
## 1: AES-128, 2: 3DES
## 3: DEs 4: AEs-192
## 5: AES-256 0: Any
################################################## #
SET NVIKEP1ENCALG 5

################################################## #
## IKE Auth algo.
## 0: Any, 1: MD5
## 2: sHA-1
################################################## #
SET NVIKEP1AUTHALG 1

################################################## #
## IKE Config Mode.
## 0: Enabled, 1: Disabled.
################################################## #
SET NVIKECONFIGMODE 0

################################################## #
## IPsec PFS DH group.
################################################## #
#SET NVPFSDHGRP 14

################################################## #
## IPsec Encryption Algo.
## 1: AES-128, 2: 3DES
## 3: DEs 4: AEs-192
## 5: AES-256 6: None
## 0: Any
################################################## #
SET NVIKEP2ENCALG 5

################################################## #
## IPsec Authentication Algo.
## 0: Any, 1: MD5
## 2: sHA-1
################################################## #
SET NVIKEP2AUTHALG 1

################################################## #
## Specifies the IKE SA lifetime in seconds
################################################## #
SET NVIKEP1LIFESEC 86400
SET NVIKEP2LIFESEC 86400

################################################## #
## Protected Network.
################################################## #

#SET NVIPSECSUBNET 0.0.0.0/0, 0.0.0.0/0
################################################## #
## IKE Over TCP.
## 0: Never, 1: Auto
## 2: Always
################################################## #
SET NVIKEOVERTCP 1

################################################## #
## Craft access
## 0: Enabled, 1: only view option is available?
################################################## #

SET PROCSTAT 0
################################################## #
## VPN craft access
## 0: disabled, 1: view only
## 2: View and edit.
################################################## #

SET VPNPROC 2
################################################## #
## Call Server address
################################################## #

SET MCIPADD 192.168.0.4

################################################## #
## craft access code
################################################## #

SET PROCPSWD 27238

################################################## #
## VPN craft access code
################################################## #

# END

RE: 9608 VPN issue - Invalid Configuration

What do you expect from us? You have to compare the phone config with the IPSec config of your firewall, read some logs and check where the error is.

RE: 9608 VPN issue - Invalid Configuration

(OP)
Is there a way I can access the log files of the phone to see the detailed error?

RE: 9608 VPN issue - Invalid Configuration

You have to check the log files of the firewall.

RE: 9608 VPN issue - Invalid Configuration

Derfloh is correct. look at your firewall logs

RE: 9608 VPN issue - Invalid Configuration

(OP)
Firewall is reporting that the PSK is not matching (by the sounds of below).

We have triple check both ends , and Pre-shared key is correct!

We haven't had this issues with the 10+ IPsec active VPNs we established.

Any ideas what could be different with the 9608?

"VPN-dynamic"[3] 195.XX.XX.XX #4824: sending notification PAYLOAD_MALFORMED to 195.XX.XX.XX:2070
"VPN-dynamic"[3] 195.XX.XX.XX #4824: next payload type of ISAKMP Identification Payload has an unknown value: 228
"VPN-dynamic"[3] 195.XX.XX.XX #4824: probable authentication failure (mismatch of preshared secrets?): malformed payload in packet
| payload malformed after IV
"VPN-dynamic"[3] 195.XX.XX.XX #4824: sending notification PAYLOAD_MALFORMED to 195.XX.XX.XX:2070
"VPN-dynamic"[3] 195.XX.XX.XX #4824: ignoring informational payload, type INVALID_COOKIE msgid=00000000
"VPN-dynamic"[3] 195.XX.XX.XX #4824: received and ignored informational message
"VPN-dynamic"[3] 195.XX.XX.XX #4824: max number of retransmissions (2) reached STATE_MAIN_R2
"VPN-dynamic"[3] 195.XX.XX.XX: deleting connection "VPN-dynamic" instance with peer 195.XX.XX.XX {isakmp=#0/ipsec=#0}

RE: 9608 VPN issue - Invalid Configuration

There have been issues in the past with the VPN software on the 9608 which has been resolved in later releases. What version are you running?

RE: 9608 VPN issue - Invalid Configuration

(OP)
Currently running version 9.1.9

RE: 9608 VPN issue - Invalid Configuration

Are you set to PSK but then using a userlogin, wouldn't this mean you need PSK with XAuth?

| ACSS SME |

RE: 9608 VPN issue - Invalid Configuration

(OP)
I found this VPN config template from another tread, and just adapted it for our environment.

Yes, we have configured the firewall for PSK and also the phone, and I just left the Xauth login in the config file, as assumed the phone would ignore as the PSK parameter is invoked "SET NVVPNAUTHTYPE 3"

Going to hash it out and try again...will advise shortly. thanks

RE: 9608 VPN issue - Invalid Configuration

(OP)
Done the following ;

Hashed out the Xauth username (### SET NVVPNUSER mscep1) and password (### SET NVVPNPSWD mscep1) parameters from 96xxvpn.txt, clearing the phone, re-adding group (876), connecting phone to LAN to grab updated VPN config file (96xxvpn.txt).

Then connected it back to the router connected to outside line (separate to firewall), and still getting the same errors in the firewall

"VPN-dynamic"[3] 195.XX.XX.XX #4824: sending notification PAYLOAD_MALFORMED to 195.XX.XX.XX:2070
"VPN-dynamic"[3] 195.XX.XX.XX #4824: next payload type of ISAKMP Identification Payload has an unknown value: 228
"VPN-dynamic"[3] 195.XX.XX.XX #4824: probable authentication failure (mismatch of preshared secrets?): malformed payload in packet

RE: 9608 VPN issue - Invalid Configuration

(OP)
Upgrading the firmware to 9.1.10 appears to have corrected that issue with Phase 1 negotiations , no getting issues with Phase 2.

Anyone have any ideas?

sending encrypted notification UNSUPPORTED_EXCHANGE_TYPE to 195.XX.XX.XX:4500
received MODECFG message when in state STATE_MAIN_R3, and we aren't xauth client

RE: 9608 VPN issue - Invalid Configuration

(OP)
Hi,
Also does anyone know if this phone supports L2TP?

Thanks

RE: 9608 VPN issue - Invalid Configuration

Which brand and model of firewall are you using?

"Trying is the first step to failure..." - Homer

RE: 9608 VPN issue - Invalid Configuration

(OP)
It's a all-in-one firewall , vpn, anti-virus managed proprietary linux appliance. It isn't branded per-se

RE: 9608 VPN issue - Invalid Configuration

Sure you're using MD5 as Auth encryption? Probably it's using SHA1.

"Trying is the first step to failure..." - Homer

RE: 9608 VPN issue - Invalid Configuration

(OP)
Hi,

Yes, I have checked on the actual phone, the Auth encryption is MD5 for both IKE and And IPsec, see below

################################################## #
## IKE Auth algo.
## 0: Any, 1: MD5
## 2: sHA-1
################################################## #
SET NVIKEP1AUTHALG 1

################################################## #
## IPsec Authentication Algo.
## 0: Any, 1: MD5
## 2: sHA-1
################################################## #
SET NVIKEP2AUTHALG 1

This post suggests that the error is related to IP addresses...https://supportforums.cisco.com/discussion/1246850...
"The problem is that if your remote network is using the same subnet as the local network then the router is not capable of knowing when it is supposed to send a request to the LAN or the Remote end"

The Protected Network subnet is different on the phone to firewall. I have also tried changing the protected network to 0.0.0.0/24 same error, which leads me to believe not ip related ...I have hit a brick wall :(


RE: 9608 VPN issue - Invalid Configuration

Yeah, but are you using MD5 on the VPN router, this wouldn't be recommended, you should use at least SHA-1 for security reasons.

"Trying is the first step to failure..." - Homer

RE: 9608 VPN issue - Invalid Configuration

(OP)
Hi,

Changing to SHA-1 hasn't made any difference.

RE: 9608 VPN issue - Invalid Configuration

"malformed payload in packet" usually means that the PSK is incorrect.
You need to make sure that all the settings are correct on both ends.

As you have a "no-name" VPN concentrator there is no Application Notes on how to get this working and it's hard to verify that you've done all the steps correctly.

"Trying is the first step to failure..." - Homer

RE: 9608 VPN issue - Invalid Configuration

(OP)
Hi, appreciate your comments. Just to clarify We managed to resolved the "Malformed payload" by updating the firmware on the phone.

The issue we have now is;
sending encrypted notification UNSUPPORTED_EXCHANGE_TYPE to 195.XX.XX.XX:4500
received MODECFG message when in state STATE_MAIN_R3, and we aren't xauth client

Any ideas what this could be?

RE: 9608 VPN issue - Invalid Configuration

I'm guessing one end is set for Xauth and the other is not.

Have you tried Google on the error messages, you get a lot of hits there as they are standard messages.
I could Google it for you but should go faster without the middleman =)

"Trying is the first step to failure..." - Homer

RE: 9608 VPN issue - Invalid Configuration

(OP)
Yeap, not lazy like that lol...Googled straight off - see my post my post above - 25 Apr 17 11:19, which I think you skipped over :) see below

I have setup a few VPNs in my time (with this appliance) and never had an issue.

This Avaya phone is hard work to setup, and i'm just wondering if anyone actually managed to setup the VPN with any other branded VPN devices it lists on the documentation. In fact the lack of logging and generic errors on the phone makes it near impossible to troubleshoot.

QUOTED 25 Apr 17 11:19,
Yes, I have checked on the actual phone, the Auth encryption is MD5 for both IKE and And IPsec, see below

################################################## #
## IKE Auth algo.
## 0: Any, 1: MD5
## 2: sHA-1
################################################## #
SET NVIKEP1AUTHALG 1

################################################## #
## IPsec Authentication Algo.
## 0: Any, 1: MD5
## 2: sHA-1
################################################## #
SET NVIKEP2AUTHALG 1

This post suggests that the error is related to IP addresses...Link
.
"The problem is that if your remote network is using the same subnet as the local network then the router is not capable of knowing when it is supposed to send a request to the LAN or the Remote end"

The Protected Network subnet is different on the phone to firewall. I have also tried changing the protected network to 0.0.0.0/24 same error, which leads me to believe not ip related ...I have hit a brick wall :(

RE: 9608 VPN issue - Invalid Configuration

I've set this up against a bunch of other appliances, but it's pretty hard to guess where it goes wrong without seeing the configuration for both ends.
I usually used PSK with Xauth on my installations.

"Trying is the first step to failure..." - Homer

RE: 9608 VPN issue - Invalid Configuration

(OP)
Hi,
Would you be able to share your working config?

Do you enter the config directly into the phone or use the 46xxsettings method?

Thanks

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Resources

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close