Troubleshooting Cisco Unity

Connection
BRKUCC-1900

Bryan Shapess

2
Agenda
• Troubleshooting Methodology
• Troubleshooting Tools
• CUC Traces
• Case Studies
– Message Waiting Indicator
– Visual Voicemail
– Single Inbox Message Delivery

• Note: Versions used for this presentation are Unity Connection 9.1.
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Troubleshooting Methodology
Where Does Unity Connection Fit?
• Unity Connection Integrates with:
– CUCM via SIP or SCCP
– Third party PBX via PIMG/TIMG
• Can optionally use External LDAP Directory
• Can optionally store copy of messages in External Message Store

Message Unity
Store Directory

Telephony
(CUCM, SIP, PIMG)
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Start by Gathering Information
• Detailed problem description
– Exact time (last week, yesterday, this morning around 9:00)
– What was the user doing? (What options did the user select or what digits were
entered?)
– What did the user hear?
– What did the user expect to happen?
• Determine frequency
– Is it reproducible?
– How often does it occur?
– Is there a Pattern?
• Isolate separate issues and prioritize

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Examine Available Information

• Use tools such as the application syslog to search for errors

• Gather and examine traces

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Prepare for Next Occurrence
• Configure macro and/or micro traces
• Notify user(s) of what information to capture
– Time of issue
– What they were trying to do
– Digits entered
– Recording(s) heard
• Be prepared to collect traces

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Troubleshooting Tools
Troubleshooting Tools
• Remote Port Status Monitor
• Cisco Utilities Data Link for Informix (CUDLI)
• Real Time Monitoring Tool (RTMT)
• Packet Capture
• EWS Editor

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Remote Port Status Monitor
• Available on
CiscoUnityTools.com
• Alternates color with
each new call

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Cisco Utilities Data Link for Informix (CUDLI)
• Found on www.ciscounitytools.com
• Read-only access to UnityDirDB, UnityRptDB, UnityDynDB and UnityMBX1DB
• Provides additional information on tables and columns within those tables

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
CUDLI: Column Info

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
CUDLI: Mouse Over for More Information

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
CUDLI: Options

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
CUDLI: Search

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
CUDLI: Search

Search Results

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
CUDLI: Has a Built-In Query Builder

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
RTMT
• Downloaded from Cisco Unity Connection Administration  System Settings 
Plug-ins
• Provides performance monitoring data (CPU, memory, disk usage, etc.)
• Provides alerting capabilities
• Syslog Viewer provides logging information for the server similar to Windows
Event Viewer
• Trace and Log Central is used to collect trace files

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Syslog Viewer Sample

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Syslog Viewer Sample

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
RTMT: Collect Files
Select services as
needed.

• Found under Trace &

Log Central -> Collect
Files
• Refer to macro or micro
trace table in
troubleshooting guide for
traces to retrieve

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
RTMT: Collect Files Cont’d

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
RTMT: Collect Files
• Select Date/Time Range
• Specify the Download File Directory
• Select if you want to Zip Files

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
RTMT: Scheduled Trace Collection

• Same look as manual

Trace Collection
• Allows for Automatic
Trace Collection

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
RTMT: Scheduled Trace Collection

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
RTMT: Scheduled Trace Collection

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
RTMT: Scheduled Trace Collection

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
RTMT: Scheduled Trace Collection

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Packet Capture
Why would you use a packet capture?
• To see packet headers and payload
• Help diagnose issues when traces don’t seem to help
• Provide product neutral data
• Make sure packets are making it to intended destination

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Packet Capture
• Built-in to Unity Connection admin cli
• Syntax:
– utils network capture [page] [numeric] [file fname] [count num] [size bytes] [src
addr] [dest addr] [port num] [host protocol addr]

• Example syntax to capture SMTP traffic to a file called CUC.cap:

– utils network capture file CUC count 100000 size all port 25

• When finished press <ctrl> + c to stop capture

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Packet Capture
• Can download packet
capture from RTMT or
from CLI
• Must analyze packet
capture in 3rd party
tool

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Packet Capture
• To view available capture files from CLI
admin:file list activelog platform/cli
cuc.cap
dir count = 0, file count = 1

• To download capture file from CLI

admin:file get activelog platform/cli/cuc.cap
Please wait while the system is gathering files info ...done.
Sub-directories were not traversed.
Number of files affected: 1
Total size in Bytes: 1668
Total size in Kbytes: 1.6289062
Would you like to proceed [y/n]?
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Packet Capture
Wireshark

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
EWS Editor
• A Microsoft tool which simulates the EWS connection to Exchange
• Helps diagnose Unity Connection single inbox connection issues to Exchange
• https://ewseditor.codeplex.com/
• Not TAC supported

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
EWS Editor
• Service URL:
– http(s)://<Exchange CAS
Server>/EWS/Exchange.asmx
• Requested Exchange Version:
– Exchange2007_SP1
• Credentials:
– Same credentials entered in CUC for
Unified Messaging Service
• Impersonation:
– Email Address of user’s mailbox

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
EWS Editor
• Will receive this error if you
are:
– Using self signed certificate on
Exchange
– Don’t have root certificate
trusted on workstation

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
EWS Editor

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
EWS Editor

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
CUC Traces
Macro Traces: Unity Connection
• A collection of preselected micro
traces
• Used to troubleshoot general areas
of functionality

Select area where

you need traces

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Micro Traces: Unity Connection
• Have to select one micro trace
grouping at a time
• Used for more in depth
troubleshooting
Select
Component

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Micro Traces: Unity Connection

Check
Individual
Tracing Levels

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
What Traces Should I Enable?
• Diagnostic Traces section in Troubleshooting Guide -
http://www.cisco.com/en/US/docs/voice_ip_comm/connection/9x/troubleshooting
/guide/9xcuctsg010.html

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
What Traces Should I Enable?
• When in doubt enable the following
Macro Traces:
– Call Flow Diagnostics
– Call Control (Miu) Traces
– Conversation Traces
• Collect Traces from RTMT:
– Connection Conversation Manager

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Unity Connection Traces
• UC traces end with extension .uc
• Plain text
• Format:
– 16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
UC Trace Deciphered
• Format is:
• Time | Thread, Port, Call GUID, Component, Level, Function or Process

• Trace shows:
–16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
• Event occurred at 4:13 PM

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
UC Trace Deciphered
• Format is:
• Time | Thread, Port, Call GUID, Component, Level, Function or Process

• Trace shows:
–16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
• This process uses thread 29183

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
UC Trace Deciphered
• Format is:
• Time | Thread, Port, Call GUID, Component, Level, Function or Process

• Trace shows:
–16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
• Call involved Port 2 on Port Group CUCM-SIP-1

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
UC Trace Deciphered
• Format is:
• Time | Thread, Port, Call GUID, Component, Level, Function or Process

• Trace shows:
– 16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
• The Call GUID is a unique identifier; using a grep tool, you can filter all
associated traces for this call

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
UC Trace Deciphered
• Format is:
• Date Time | Thread, Port, Call GUID, Component, Level, Function or Process

• Trace shows:
–16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
• Low Level (1) Conversation Engine (CDE) trace

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
UC Trace Deciphered
• Format is:
• Date Time | Thread, Port, Call GUID, Component, Level, Function or Process

• Trace shows:
–16:13:57.240 |29183,CUCM-SIP-1-
002,EE862E7085FA47719E2349568CBE8332,CDE,1,aDtmfAccessId = 1008
• The DTMF Access ID (extension) is 1008. This happens to be the calling
number for this call.

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Case Studies

53
Case Study 1 – MWI Failure
• Problem Description:
– A user reports that the red light on the phone is off but when they called Unity
Connection it said they had a new message

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Case Study 1 – MWI Failure
• What trace to enable?
– Macro Traces
• Call Control (Miu) Traces
• Traces for MWI problems

• What traces to retrieve?

– Connection Conversation Manager
– Connection Notifier

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Case Study 1 – MWI Failure
• Open diag_CuNotifier and search for Failed.

14:43:58.086 |24630,,,Notifier,12,Bryan Shapess: MWI (ON) failed. Details=Failed

MWI [user='Bryan Shapess' extension='1000' type=MWI_ON mwiType=0=Port
phoneSystem=CUCM-SIP (bd22ed0b-83ce-4871-a67f-ac3c530ad233)] Error=No
MWI ports on switch
BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Case Study 1 – MWI Failure

View with list

of phone
systems

Phone
Systems with
port count

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Case Study 1 – MWI Failure

View with
subscribers’
and their
phone system

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Case Study 1 – MWI Failure

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Case Study 1 – MWI Failure
• Change phone system to proper phone system.

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Case Study 2 – Visual Voicemail Failure
• Problem Description:
When trying to play a message using Visual Voicemail you receive an error “No Line
Available To Play Message”. Issue happened around 8:55pm

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Case Study 2 – Visual Voicemail Failure

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Case Study 2 – Visual Voicemail Failure
• What trace to enable?
– Macro Traces
• Call Control (Miu) Traces
– Micro Traces
• VMWS (All)
– Phone Service debug logging
– CallManager Detailed traces

• What traces to retrieve?

– Connection Conversation Manager
– Connection Tomcat Application
– Cisco Tomcat (Second Page)
– Phone Console Logs
– CallManager Detailed SDI Logs

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Case Study 2 – Visual Voicemail Failure
• Set Phone Service VisualVoicemail log_level to debug
• Collect console logs from phone web page

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Case Study 2 – Visual Voicemail Failure
• Find calls to Reverse Trap Number
• Locate call which matches time user experienced problem

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Case Study 2 – Visual Voicemail Failure

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Case Study 2 – Visual Voicemail Failure
• Failure caused by defect CSCuj81059
– Symptom:
• On RT phones, having phone firmware 9.2.2 onwards, when line having a busy line/shared line the
VVM does not play and gives the following error: "No line available to play message“
– Resolved:
• 8.6.2SU4
• 9.1.2ES27

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Case Study 3 – Single Inbox Failure
• Problem Description:
A user reports that when they get a new message it doesn’t show up in their Outlook
Inbox

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Case Study 3 – Single Inbox Failure
• What traces to enable?
– Macro: Single Inbox Trace
– Micro: EWS (All)
• What traces to retrieve?
– Connection Mailbox Sync

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Case Study 3 – Single Inbox Failure
• Open trace file CuMbxSync

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Case Study 3 – Single Inbox Failure

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Case Study 3 – Single Inbox Failure

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Case Study 3 – Single Inbox Failure

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Case Study 3 – Single Inbox Failure
• Check Exchange for users whose mailbox is over quota

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Case Study 3 – Single Inbox Failure
• To better understand single inbox messages, review single inbox annotated logs
• http://www.ciscounitytools.com/Training/Connection/CUC8_5.html

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Complete Your Online Session Evaluation
• Give us your feedback and you
could win fabulous prizes. Winners
announced daily.
• Complete your session evaluation
through the Cisco Live mobile app
or visit one of the interactive kiosks
located throughout the convention
center.

Don’t forget: Cisco Live sessions will be available

for viewing on-demand after the event at
CiscoLive.com/Online

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Continue Your Education
• Demos in the Cisco Campus
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings

BRKUCC-1900 © 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 77