Troubleshooting Mcrnc

WCDMA RAN, Rel. RU50 andRU50 EP1, OperatingDocumentation, Issue 05

TroubleshootingMulticontroller RNC DN0976768Issue 02BApproval Date 2015-02-23

NokiaNetworks

The information in this document applies solely to the hardware/software product (Product) specifiedherein,andonlyasspecifiedherein.

ThisdocumentisintendedforusebyNokiaSolutionsandNetworks'customers(You)only,anditmaynotbeusedexceptforthepurposesdefinedintheagreementbetweenYouandNokiaSolutionsandNetworks(Agreement)underwhich thisdocument isdistributed.Nopartof thisdocumentmaybeused,copied,reproduced,modified or transmitted in any form ormeanswithout the priorwritten permission ofNokiaSolutionsandNetworks. If youhavenotentered intoanAgreementapplicable to theProduct,or if thatAgreementhasexpiredorhasbeenterminated,Youmaynotusethisdocument inanymannerandYouareobligedtoreturnittoNokiaSolutionsandNetworksanddestroyordeleteanycopiesthereof.

The document has been prepared to be used by professional and properly trained personnel, andYouassumefullresponsibilitywhenusingit.NokiaSolutionsandNetworkswelcomeYourcommentsaspartoftheprocessofcontinuousdevelopmentandimprovementofthedocumentation.

This document and its contents are provided as a convenience to You. Any information or statementsconcerningthesuitability,capacity, fitnessforpurposeorperformanceof theProductaregivensolelyonanasisandasavailablebasisinthisdocument,andNokiaSolutionsandNetworksreservestherighttochangeanysuchinformationandstatementswithoutnotice.NokiaSolutionsandNetworkshasmadeallreasonableeffortstoensurethatthecontentofthisdocumentisadequateandfreeofmaterialerrorsandomissions,andNokiaSolutionsandNetworkswill correcterrors thatYou identify in thisdocument.But,NokiaSolutionsandNetworks'totalliabilityforanyerrorsinthedocumentisstrictlylimitedtothecorrectionofsucherror(s).NokiaSolutionsandNetworksdoesnotwarrantthattheuseofthesoftwareintheProductwillbeuninterruptedorerror-free.

NOWARRANTYOFANYKIND,EITHEREXPRESSORIMPLIED, INCLUDINGBUTNOTLIMITEDTOANY WARRANTY OF AVAILABILITY, ACCURACY, RELIABILITY, TITLE, NON-INFRINGEMENT,MERCHANTABILITYORFITNESSFORAPARTICULARPURPOSE, ISMADE INRELATIONTOTHECONTENT OF THIS DOCUMENT. IN NO EVENT WILL NOKIA SOLUTIONS AND NETWORKS BELIABLE FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT,INCIDENTALOR CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT NOT LIMITED TO LOSSOFPROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA THAT MAYARISEFROMTHEUSEOFTHISDOCUMENTORTHEINFORMATIONINIT,EVENINTHECASEOFERRORSINOROMISSIONSFROMTHISDOCUMENTORITSCONTENT.

ThisdocumentisNokiaSolutionsandNetworksproprietaryandconfidentialinformation,whichmaynotbedistributed or disclosed to any third parties without the prior written consent of Nokia Solutions andNetworks.

Nokia isa registered trademarkofNokiaCorporation.Otherproductnamesmentioned in thisdocumentmaybetrademarksoftheirrespectiveowners,andtheyarementionedforidentificationpurposesonly.

Copyright2015NokiaSolutionsandNetworks.Allrightsreserved.

f Important Notice on Product Safety Thisproductmaypresentsafetyrisksduetolaser,electricity,heat,andothersourcesofdanger.

Only trained and qualified personnel may install, operate, maintain or otherwise handle thisproductandonlyafterhavingcarefullyreadthesafetyinformationapplicabletothisproduct.

The safety information is provided in the Safety Information section in the Legal, Safety andEnvironmentalInformationpartofthisdocumentordocumentationset.

Nokia Solutions and Networks is continually striving to reduce the adverse environmental effects of itsproductsandservices.Wewouldliketoencourageyouasourcustomersanduserstojoinusinworkingtowardsacleaner,saferenvironment.Pleaserecycleproductpackagingandfollowtherecommendationsforpoweruseandproperdisposalofourproductsandtheircomponents.

IfyoushouldhavequestionsregardingourEnvironmentalPolicyoranyoftheenvironmentalservicesweoffer,pleasecontactusatNokiaSolutionsandNetworksforanyadditionalinformation.

TroubleshootingMulticontrollerRNC

2 DN0976768 Issue:02B

Table of ContentsThisdocumenthas71pages

Summaryofchanges..................................................................... 7

1 OverviewofMulticontrollerRNCTroubleshooting......................... 81.1 TroubleshootingMulticontrollerRNCrecommendations................81.2 Informationsourcesinfaultsituations............................................91.3 Problemtypes.............................................................................. 101.4 Generictroubleshootingprocedure.............................................. 11 2 ReportingproblemtoNSN........................................................... 14 3 Symptomsdatacollection............................................................ 193.1 Standardsymptomsreport...........................................................193.2 Collectingstandardsymptomreports...........................................233.3 Listingthestandardsymptomreports.......................................... 283.4 Copyingstandardsymptomsreportstoremotemachine.............293.5 Deletingthestandardsymptomreports....................................... 30 4 Genericsoftwaretroubleshootinginstructions............................. 324.1 Displayingallblackboxfiles......................................................... 324.2 Viewingactivealarmsandalarmshistory.................................... 334.3 Collectingcoredumpfileinformation........................................... 34 5 Configurationmanagementtroubleshooting................................ 365.1 Savingconfigurationsnapshotfails..............................................365.1.1 Description................................................................................... 365.1.2 Symptoms.................................................................................... 365.1.3 Recoveryprocedures................................................................... 36 6 Softwaremanagementtroubleshooting....................................... 376.1 Installationofanewsoftwaredeliveryfails.................................. 376.1.1 Description................................................................................... 376.1.2 Symptoms.................................................................................... 376.1.3 Recoveryprocedures................................................................... 376.2 Activationofanewsoftwaredeliveryfails....................................376.2.1 Description................................................................................... 376.2.2 Symptoms.................................................................................... 386.2.3 Recoveryprocedures................................................................... 386.3 Troubleshootingsoftwareconfigurationmanagement................. 386.3.1 Executingpre-downloadscriptfails..............................................386.3.2 ParsingtargetBD.xmlfails............................................................396.3.3 Executingpre-activationscriptfails..............................................396.3.4 UpgradingeSWfails.................................................................... 406.3.5 Alarm2518-NOVALIDFALLBACKCOPYFORDEFAULT

PACKAGEistriggered................................................................. 41


Issue:02B DN0976768 3

7 Networkingtroubleshooting..........................................................437.1 Packetlossincertaintrafficflowsunexpectedlyhigh.................. 437.1.1 Description................................................................................... 437.1.2 Symptoms.................................................................................... 437.1.3 Recoveryprocedures................................................................... 437.2 MultihopBFDsessionsarenotestablished................................ 467.2.1 Description................................................................................... 467.2.2 Symptoms.................................................................................... 467.2.3 Recoveryprocedures................................................................... 467.3 OSPFisnotworkingproperly...................................................... 487.3.1 Description................................................................................... 487.3.2 Symptoms.................................................................................... 487.3.3 Recoveryprocedures................................................................... 497.4 IPsignalinglinkactivationfails.....................................................527.5 Thestateofallsubsystemsintheremotenetworkelementis

unavailable(UA)...........................................................................53 8 Hardwaretroubleshooting............................................................ 548.1 Notrafficdetected........................................................................ 548.1.1 Description................................................................................... 548.1.2 Symptoms.................................................................................... 548.1.3 Recoveryprocedure.....................................................................548.2 SFPportsaddedforUSPUsandCSPUsdonotshowcorrectly

theiroperationalstatenoralarmsareraised................................558.3 TroubleshootingwithmcJANEtool.............................................. 56 9 Performancemanagementtroubleshooting................................. 699.1 ThresholdmonitoringalarmisnotsenttoNetAct........................ 699.2 TransportandHWmeasurementmanagementinmcRNC..........709.2.1 McRNCmeasurementmanagementconcepts............................ 709.2.2 McRNCmeasurementmanagementcommands......................... 71


4 DN0976768 Issue:02B

List of FiguresFigure1 ForcingchangeoftheSPFpoststate................................................ 64Figure2 Exampleusageofthecpcommand................................................... 65Figure3 Checkingtheportmonitoring..............................................................66Figure4 Removingportmonitoring.................................................................. 67


Issue:02B DN0976768 5

List of TablesTable1 NSNCaseType..................................................................................15Table2 NSNProblemPriority......................................................................... 16Table3 Supportedplugins...............................................................................19Table4 Parametersrelatedtosave symptom-reportcommand.............. 24Table5 Mappingbetweenpip-markandQueueID......................................44Table6 PriorityvaluesfordifferentDSCPcodes............................................ 45


6 DN0976768 Issue:02B

Summary of changesChangesbetweendocumentissuesarecumulative.Therefore,thelatestdocumentissuecontainsallchangesmadetopreviousissues.

Changes between issues 02A(2014-05-23, RU40) and 02B(2015-02-23,RU40)Newsubchapterhasbeenadded.Troubleshooting software configurationmanagement

Changes between issues 02(2013-09-15, RU40) and 02A(2014-05-23,RU40)OSPF is not working properlyhasbeenupdated.ThefeaturemanagementnamehasbeenchangedtoOSPFForRedundancy.

Changes between issues 01D (2012-11-23, RU30) and 02(2013-09-15,RU40)Activation of a new software delivery fails

Commandshavebeenupdated.

Installation of a new software delivery fails


Displaying all blackbox files

InformationonpermissionsrequiredforexecutingthecommandinthissectionisaddedtoBeforeyoustartsection.

Collecting core dump file information


TroubleshootingMulticontrollerRNC Summaryofchanges

Issue:02B DN0976768 7

1 Overview of Multicontroller RNCTroubleshooting

1.1 Troubleshooting Multicontroller RNCrecommendationsIfyouhaveacontractwithNokiafortheoperationandmaintenanceofthenetwork(orsomeotheragreement),theactionsyouneedtotakeinafaultsituationmaybedifferentfromtheonessuggestedinthetroubleshootinginstructions.Ifthegeneralprinciplesareinconflictwiththeoperationandmaintenancecontractoranyothercontract,carryoutactionsasagreedinthecontract.

TheoperationandmaintenancepersonnelthatcarryouttroubleshootingshouldbefamiliarwiththehardwareandsoftwareofNokianetworkelements.

Electrostatic precautionsWhenhandlingplug-inunits,itisimportanttouseElectrostaticPrecautions(ESP).Thismeansthatyoumustbeearthedtoequipmentracksusinganapprovedwriststrapandconnectinglead.ApprovedESPequipmentmakesaresistiveconnectiontoensurethesafetyofthepersonnelandtopreventasuddenstaticdischargeduringconnectiontotheearthingpoint.

Security proceduresYouarerecommendedtoestablishsecurityproceduresatyoursitetoensureappropriatestaffandterminalaccesstothepersonnel.

Disaster recovery planEstablishadisasterrecoveryplantohelpthepersonneltodealwithemergencysituations.Rememberthatemergencysituationscanbebestavoidedbydetectingabnormalconditionsearly.Adisasterrecoveryplanshouldcovervariousdisasterscenariosanddisasterrecoveryproceduresforpersonnel.Theoperationandmaintenancepersonnelshouldalsobeabletocontactthepersonswhoarecapableofdealingwiththeprobleminquestion.Therefore,eachsiteshouldhaveanescalationplanavailablewithappropriatecontactinformation.

Escalation planAnescalationplanofferscontactlistsofinternalandexternalsupportpersonnelandservicesavailabletotackleproblems.Itshouldcontaininformationonwhotocontactandinwhatkindofsituations,forexample,airconditioning,powerback-upsystemandNokiaEmergency/HelpDesknumbers.

Preventive maintenancePerformpreventivemaintenanceroutinesonaregularbasis.Forexample,carryoutregularalarmandunitstatesurveillance.

OverviewofMulticontrollerRNCTroubleshooting TroubleshootingMulticontrollerRNC

8 DN0976768 Issue:02B

SafecopyingTakingregularsafecopiesofthesoftwareanddatabasesofthenetworkelementensuresthatyouhaveafunctionalcopyofthesoftwarewhichyoucanuseifthereareproblemswiththesoftwareorhardwareofthenetworkelement.Howoftenyouneedtotakesafecopiesdependsonthesizeandtypeofthenetworkelement.

Performance monitoringThepurposeofperformancemonitoringistomeasuretheoverallqualityofthesystem.Performancemonitoringcanhelpyoutodetectverylowrateorintermittentproblemsandpossibledegradationofsomepartofthesystem.

Theperformancemonitoringparametersfornetworkelementsofdifferentkindsandsizesvary.

DocumentationEstablishaprocedureforkeepingthedocumentationup-to-dateandmakesurethattheoperationandmaintenancepersonnelhaveaccesstoallrelevantexternalandinternaldocuments.

Network element diaryItisrecommendedthatyoumaintainanetworkelementdiary.Thediaryshouldbenetworkelement-specific,butyoucanstoreitintheOperationandMaintenanceCentreifthenetworkelementisnotusuallymanned.

Startfillinginthenetworkelementdiaryalreadywhenthenetworkelementisbeingsetupandinstalled.

Youarerecommendedtorecordthefollowingeventsinthenetworkelementdiary:

Hardwarechanges Softwareandhardwareupdates(forexample,changenotesandcorrection

deliveries) Essentialmodificationstotheconfigurationorroutinginthenetworkelement Safecopying Operationalfailures Anyotherrelevantinformation

Anetworkelementdiarycanprovideusefulinformationonthesystem'sperformanceinthepastandhintsonwhatmightcausethecurrentproblems.

1.2 Information sources in fault situationsUseatleastthefollowinginformationsourceswhencarryingouttroubleshooting.

AlarmsAlarmsaretheprimarysourceofinformationinmostsituationswheretroubleshootingisneeded.Alarmsareprintedoutonthealarmprinterand/orotherdevicesthatyouhavespecifiedforthenetworkelement.

TroubleshootingMulticontrollerRNC OverviewofMulticontrollerRNCTroubleshooting

Issue:02B DN0976768 9

Diagnosis reportsDiagnosisreportscontaindataontheplug-inunitsthatthesystemsuspectstobefaulty.Diagnosisreportsareusuallyprintedoutonthereportprinterand/orotherdevicesthatyouhavespecifiedforthenetworkelement.

Error messagesGeneralerrormessagesofthesystemtellwhythesystemcannotcarryoutatask.Theycanappearinthesupplementaryinformationfieldsofalarmsanddiagnosticprintouts,intheprintoutsofthestartingphasesmonitoredthroughaserviceterminal,andinSCLIandElementManagercommandoutputs.

Statistical reportsDifferentstatisticalreportscontainusefuldata,forexample,ontrafficonspeechandsignallingcircuits,useofservicesandloadandavailabilitymeasurements.Monitorandassessstatisticaldataregularlyasthisdatacanindicateforthcomingproblemsbeforetheyaffectthetraffic.Formoreinformation,seeperformancemanagementdocumentation.

Logs and other relevant statistical informationDifferentlogs(forexample,computerandoperatingsystemlogsandSCLIsessionreports)containusefuldatathatcanbeattachedtothefaultreportwhenyouneedNokia'helptosolveaproblem.Takethelogsfromtheunitsendingthealarmandtheunitthatistheobjectofthealarm,andifthealarmisI/Orelated,fromtheOMU.

Unit statesCheckalsotheunitstates.

1.3 Problem typesHerearesomeproblemtypesyoumayencounter.

Reproducible problemsYoucanreproducethesymptomsusingasetofactions.Reproducibleproblemscanbesolvedbynarrowingdownthepossiblecausesoftheproblemtoasinglecauseortoanumberofcausesandapplyingcorrectiveactions.Thisrequiresknowledgeofhowthesystemworksandteststoeliminatewrongconclusions.

Intermittent problemsYoucannotreproducethesymptomsconsistentlyusinganysetofactions.However,anintermittentproblemcanreproduceitselfrandomly.Insuchacase,somekindoftracingormonitoringofthesystemmayleadyoutotheoriginofthetrouble.

Ifanintermittentproblemoccursveryseldomandithasnoseriousconsequences,itmaybebesttojustignoretheproblem.Youcanalsoperformgeneralmaintenanceandseeiftheproblemdisappears.Iftheproblemoccursoccasionally,trytoconcludewhichfactorsseemtoaffectorcontributetotheappearanceoftheproblem.


10 DN0976768 Issue:02B

Several related or isolated troubles active at the same timeStudywhetherthesymptomsrelatetoeachotherornotandtrytoisolatetheproblemsifpossible.

1.4 Generic troubleshooting procedureDescriptionDependingonwhetheryouhaveacontractwithNokiaontheoperationandmaintenanceofthenetwork(orsomeotheragreement),theactionsthatyouneedtotakeinafaultsituationmaybedifferentfromtheonespresentedhere.

WhenyoususpectthataNokianetworkelementisnotperformingasitshould,carryoutatleastthefollowingchecks.

SymptomsANokianetworkelementisnotperformingasitshould.

Recovery proceduresTroubleshooting process before calling Nokia Solutions andNetworks help desk

Steps

1 Evaluate how serious the consequences of the trouble areIftheproblemhasveryseriousconsequences,youmayhavetocallforexperthelporapplyanemergencyplanimmediately.

2 Analyse the situation where the problem or failure first appearedConsiderthefollowingbeforeyoucarryoutanycorrectiveactions.

Whatistheproblem? Whereistheproblem? Whendidtheproblemoccur? Whatwerethecircumstancesthatledtotheproblem? Whatistheimpactoftheproblem(forexample,towhatextentdoesthefault

affecttheendcustomers)? Whoisresponsiblefortakingcareoftheproblem?

3 Try to eliminate the possibility of human error

Forexample,recentchangesinthesoftwareorhardwareconfigurationofthesystem(forexample,circuitsaddedorequippingchanged)arepossiblesourcesofproblems.Thechangesmayhavebeencarriedoutincorrectly.Checktheconfigurationparameters,physicalconnections,strappings,plug-inunitsandsoon,verycarefully.


Issue:02B DN0976768 11

show functional-unit unit-info Humanerrorisaverycommoncauseofproblemstherefore,checkand

double-checkeverypossibleproblemsource.Typehistory.

Afailurecanalsooccurspontaneously(forexample,theremoteendsystemmayhaveproblemsoraservicebreakdown,oraplug-inunitmayfailduetoageing).Checkthealarms,clearcodes,unitandlinkstatesandlogsasdescribedbelow.

4 Make an accurate description of the symptomsYoumaynotbeabletosolvetheproblemyourself.Adetaileddescriptionofthesituationwherethesymptomsoccurredcanhelpanexpertsolvetheproblem.Gatheralsodataonthefailureevent.Asymptomdescriptionshouldcontainallthebasicfacts,suchas:

Date,nameofthepersonwhodetectedthetrouble,phonenumberande-mailaddress

Detailsofthesystem;forexample,whatequipmentandsoftwareisinuse Descriptionofthesymptoms(alarms,errormessages,clearcodes,faultystates

oftheunitsandlinksandsoon) Anyotherrelevantinformation,forexample,logandmessagemonitoringfiles.All

datamaybevaluableeveniftheyseemirrelevantatthetime.Storethisinformationpreferablyinelectronicformat.

5 Check and analyse the alarm situation

Checkthealarmsthatarecurrentlyon.Youarerecommendedtoalsostudythealarmhistory.Displaythealarmhistorysothatitshowsallalarmeventsfromthetimeperiodwhichstartsonehourbeforetheoccurrenceoftheproblemsituation,andendsonehouraftertheproblemsituationwasover.Youshoulddisplaythealarmhistory.# show alarm activeThesystemmaysetanalarmandcancelitimmediately.Youcanfindthesealarmsinthealarmhistory.Alarmsbehavinginthiswaymayindicatethatsomepartofthesystemisabouttobreakdownoritsfunctionalityhasbeenreduced.

Checkalsothediagnosticreports.

6 Option Description

If thefaultcanbelocatedbasedonthealarmsituation

Then Carry out the appropriate maintenance actions

Ifyouhaveendedupwithmorethanoneprobableandpossiblecauseforthetrouble,changeonlyonethingatatimeotherwiseyoucannotbesureofwhichchangecorrectedthefailureorproblem.

Rememberthatrandomactionscanmakeproblemsworse.Generally,youshouldnottakeanyradicalcorrectiveactionsifyouarenotsurewhattheproblemisandwhattheconsequencesofthecorrectiveactionsare.Losingtrafficbecauseofincorrectactionsisnotwhatyouwant.


12 DN0976768 Issue:02B

7 Option Description

If youcannotlocatethefaultbasedsolelyonthealarmsituation

Then Try to narrow down the possible problem source

Analyseandcategorisesymptomsandlistpossiblecausesforthesymptoms.Sometimestherecanbeseveralrelatedorisolatedtroublesactiveatthesametime.Studywhetherthesymptomsrelatetoeachotherornot.Prioritisesymptomsandcollectfurtherfactsifneeded.

Basedontestsandyourknowledgeofthesystem,eliminatesymptomsthatarenotrelevanttothetroubleyouaretryingtosolve.Thiswayyoucanfocusonsymptomsandcausesthataremorelikelytoproduceasolutiontotheproblem.Examinewhatworksandwhatdoesnot.

However,eventhoughyoumaynotbeabletoanalysewhatthecauseofthetroubleis,youcancarryoutgeneralmaintenancetoeliminatesometrivialcausesoftroubles,suchasloosecablesandbadconnections.

8 Use measurements to trace any abnormal trendsFormoreinformation,seeperformancemanagementdocumentation.

9 Check the states of units, links and circuitsCheckunitstates.show functional-unit unit-info

10 Fill in a problem report if neededDescribetheproblemindetailintheproblemreport.Includeallrelevantinformationthatyouhaveavailablefromtheproblemsituationanddescribealsothecorrectivemeasuresthatyouhavecarriedoutaftertheproblemoccured.


Issue:02B DN0976768 13

2 Reporting problem to NSNProblemreportsareusedtocommunicateproblemsandfailurestoservicepersonnel.Reportonlyonefaultinoneproblemreport.Includetheattachmentsinthecompressedformat.

Tomaketheinvestigationofaproblemfaster,includethefollowinginformationintheproblemreport:

atitlethatgivesabriefdescriptionoftheproblem aclearandexactdescriptionoftheproblemitself problembackgroundinformation:

softwareandhardwarereleases situationinthebeginning,forexample,thefirstsymptomsoftheproblem situationaftertheproblemoccured operationsyoumadewhichpossiblycausedthefailure,forexample:hardware,

software,parameter,featureorconfigurationchanges,includingopertionsintransportnetworkandthirdpartyequipment

describeiftheproblemcanbereproducedandwhatactionsarerequiredtoreproducethisproblem

troubleshootingandrecoveryactionsthatyoumade

symptomdatareportsMostoftheMulticontrollerRNCconfigurationdataandlogsessentialforinvestigationoftheproblemcanbecollectedwiththestandardsymptompluginreportgroup-RNCandsubreport-MessageMonitor.ThisdataisalwayscollectedifProblemReportissubmittedtoNSN.TocollecttheMulticontrollerRNCstandardsymptomdata,executethefollowingSCLIcommands:

symptomdataandmessagemonitoring:save symptom-report name include group-RNCinclude subreport-MessageMonitor

basicsymptomdata:save symptom-report name include group-RNC

Formoreinstructionsonthestandardsymptomsdatacollectionframework,seeStandard symptoms data collection.Thedatamustbecollectedassoonaspossibleafteranabnormalsituationhastakenplaceandbeforeanyrecoveryactionisperformed,suchasMulticontrollerRNCrestartsorreplacinghardware.Thisisimportantbecausetheinformationstoredabouttheproblem(forexample,blackboxofacertainunit)maygetoverwrittenintheprocessoftimeorbelostbecauseofrecoveryactions.Thestandardsymptomreportgroup-RNCisexpectedtobecompletedinaround15minutesdependingonMulticontrollerRNCconfiguration.Recoveryactionscanbestarted,ifneeded,assoonassymptomreportgenerationiscompleted.Acopyofthesymptomreportcanbetransferredtolocalmachinelaterataconvenienttime.Whenyousendoutaproblemreport,makesurethatallthepossibleattachmentsareincludedintheproblemreport,toavoidunnecessaryinformationrequests.ForOMSsymptomdatacollection,seeTroubleshooting OMSdocument.

Ifpossible,includeadditionalproblemspecificsymptomdata.Forexample,problemspecificmessagemonitoringlogs,networkanalyzerinterfacetraces.

ReportingproblemtoNSN TroubleshootingMulticontrollerRNC

14 DN0976768 Issue:02B

NetActKPI/counterreportincasetheproblemconcernscertainKPI/counter Inamultivendorenvironment,includedetailedinformationontheotherthirdparty

products. NSNCaseType;Emergency,TroubleResolutionorTechnicalQuery.Fordetails,see

#c105225791/table_bjv_g12_rp NSNProblemPriority:Major,MediumorMinor.Fordetails,see

#c105225791/table_ndz_pd2_rp.

Table 1 NSNCaseType

NSN Case Type

NSN Definition

Examples

Emergency ThiscasetypeisforTotalandPartial outages.

Availablepriority-Critical.

ThiscasetypeisforcasesunderEmergencySupportservice.

Total Outage

Totallossofvoiceanddatatrafficcapability.Anunscheduledeventmustbelongerthan15seconds.

Partial Outage

Lossofgreaterthan10%oftheprovisionedcapacityfororiginationand/orterminationofvoiceand/ordatatraffic.Totallossofoneormorecriticalservices.Anunscheduledeventmustbelongerthan15seconds.

Total Outage:

AllIublinksaredown. AllWCELsareinincorrectstate. RNCspontaneousrestart(unplanned). RNCsystemrestart(planned),alllinks

aredown.

Partial Outage:

Lossof10%ormoreofvoiceordatatrafficcapacityforatleast15seconds.

Spontaneousrestartsofactivecomputerunit(s)whenthereisnoredundantunitavailabletotakeovertheservicesprovidedbytherestartedone.

Oneormorenetworkinterfacesaredownforover15seconds.

TotallossofsubscriberrelatedRNCfunctionality(ISHO,HSDPA,HSUPA,Internetbrowsing,andsoon)

RNCoperationandmaintenancefromNetActistotallylost.

Emergencycallsnotpossible(forexample,112or991).

TroubleResolution

Thiscasetypeisforproblemswhichcausedbyasuspectedoranidentifieddefect.

Availableproblempriorities-Major,Medium,andMinor.

SWdefectisidentifiedorsuspected.Acorrectionwillberequiredifthedefectisconfirmed.

HWdesigndefectissuspected.Acorrection(HWretrofitorSWchange)willberequiredifthedefectisconfirmed.

Anerrororomissionintheproducttechnicalliteratureissuspected.Adocumentationcorrectionwillberequiredifthedefectisconfirmed.

Technical Query Thiscasetypeisfortechnicalquestionsregardingdailynetworkoperationsandmaintenanceissues.

Technicalquestionsonproceduresorfeaturesthatarecoveredinthedocumentationshippedwiththeproduct.

TroubleshootingMulticontrollerRNC ReportingproblemtoNSN

Issue:02B DN0976768 15

Table 1 NSNCaseType(Cont.)

NSN Case Type

NSN Definition

Examples

AvailableproblemprioritiesMajorandMedium.

InformationrequestsonNSNproductthatwillbeusedtohelpinterfacethisproductwithathirdpartyproduct.

Table 2 NSNProblemPriority

NSN ProblemPriority

NSN Definition

Examples

Critical ProblemsunderEmergencySupportservice.

N/A

Major OnlyTotalorPartialoutageswhicharenotavoidablewithaworkaroundsolution.

N/A

Medium Lossoflessthan10%oftheprovisionedcapacityfororiginationand/orterminationofvoiceand/ordatatraffic.

TotalorPartialoutagesavoidablewithaworkaroundsolution.

Partiallossofoneormorecriticalservices.

totalormorethan10%lossofvoiceordatatrafficcapacityforatleast15seconds,avoidablewithworkaround

singlerestartofcomputerunits configurationchanges(RNW,

HWandSW)arenotworking activationofafeaturefails singleperformance

measurementisnotworkingcompletely

partiallossofsubscriberrelatedRNCfunctionality(ISHO,HSDPA,HSUPA,Internetbrowsing,andsoon)

partiallossofalarmmanagementofobjects(BTS,functionalunits)

problemswithback-up majorerrorsindocumentation,

forexample,analarmorparameterdescriptionismissingfromdocumentation

vitaldocumentsaremissingfromthedocumentationlibrary

Minor Minorfaultnotaffectingoperationorservicequality

failuresnotseriouslyaffectingtraffic

errorsincommandlinesyntaxoroutput

minorerrorsindocumentation


16 DN0976768 Issue:02B

MostoftheMulticontrollerRNCconfigurationdataandlogsessentialforprobleminvestigationcanbecollectedwiththestandardsymptompluginreportgroup-RNCandsubreport-MessageMonitor.ThisdataisalwayscollectedifProblemReportissubmittedtoNokiaSolutionsandNetworks.

TocollecttheMulticontrollerRNCstandardsymptomdata,executethefollowingSCLIcommands:

symptomdataandmessagemonitoring:save symptom-report name include group-RNCinclude subreport-MessageMonitor

basicsymptomdata:save symptom-report name include group-RNC

Formoreinstructionsonthestandardsymptomsdatacollectionframework,seeStandard symptoms data collection.

ThedatamustbecollectedassoonaspossibleafteranabnormalsituationhastakenplaceandbeforeanyrecoveryactionisperformedsuchasMulticontrollerRNCrestartsorreplacinghardware.Thisisimportantbecausetheinformationstoredabouttheproblemmaygetoverwrittenintheprocessoftimeorlostduetorecoveryactions.

Tosaveorcollectthebasicsymptomreportgroup-RNC,enterthefollowingcommand:

save symptom-report name include group-RNC includesubreport-MessageMonitor

_nokadmin@CFPU-0 [RNC-37] > save symptom-report name walle20141016 include group-RNC include subreport-MessageMonitorCFPU-0@RNC-37 [2014-10-16 13:48:57 +0200]Mode : Normal modeMax. execution time limit : 30 minutesMax. size of each part : 100 MBMax. size of full report : 350 MBReportName FileName Chunk Subreport Store---------- --------- ------ -------- -----walle20141016 /stdsymp/walle20141016_0.tar Part 1 ipmgmt CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 MessageMonitor CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncipconfig CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncrnw CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncsignaling CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rnchw CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncinfo CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncmon CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncuplane CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rnchas CFPU-0

TroubleshootingMulticontrollerRNC ReportingproblemtoNSN

Issue:02B DN0976768 17

walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncalarm CFPU-0

Successfully collected symptom report data for 11 out of 11 subreports

Thestandardsymptomreportgroup-RNCisexpectedtobecompletedinaround15minutesdependingonMulticontrollerRNCconfiguration.

g Whenusingthe"to"and"from"optionswiththestandardsymptomreportgroup-RNC,thesyslog,syslog.1filesarenotfilteredaccordingtothetimerangeprovided.Thetimerangeisnotapplicabletosyslogandsyslog.1files.

Recoveryactionscanbestarted,ifneeded,assoonassymptomreportgenerationiscompleted.Copyofthesymptomreportcanbetransferredtolocalmachinelaterataconvenienttime.

Whenyousendoutaproblemreport,makesurethatallthepossibleattachmentsareincludedintheproblemreport,toavoidunnecessaryinformationrequests.


18 DN0976768 Issue:02B

3 Symptoms data collection

3.1 Standard symptoms reportStandardsymptomreportisaframeworkusedtocollectsymptomdatafromMulticontrollerRNCtosupporttroubleshootingofproblems.Theframeworkcollectsstandardsymptomsdatabyrunningindividualormultipleplugins.Someofthepluginsareputintogroups,thusitispossibletocollectthesymptomdataonthegroupleveltoo.Theframeworkallowseasyenhancementwithadditionalplugins,iftheavailablestandardpluginsarenotsufficient.Thecollectedsymptomsdataisstoredin/mnt/backup/stdsympdirectory.

Thefollowingtableprovidesdetailsaboutstandardpluginsandtheirfunctionality:

Table 3 Supportedplugins

Plugin Name

Description

group-RNC ItcollectsMulticontrollerRNCconfigurationdataandlogsessentialforinvestigationoftheproblem.

w NOTICE:ThisdatamustalwaysbecollectedifProblemReportissubmittedtoNokia.Thisisthereportgroupcontainingthefollowingsub-reports:subreport-rnchw subreport-rncinfo subreport-rncipconfig subreport-rncsignaling subreport-rncrnw subreport-rnchas subreport-rncalarmsubreport-rncuplanesubreport-rncmon

group-Backupandrestore

Itcollectsthefollowingbackupandrestore-relatedlogs:

subreport-clusterinfo subreport-syslogsubreport-corefiles subreport-processinfosubreport-clusterstatus

group-Networking Itcollectsthefollowinggeneralstatisticsofthenetwork:

subreport-clusterinfo subreport-osinfosubreport-processinfo subreport-clusterstatussubreport-syslog subreport-corefilessubreport-alarmsinfo subreport-debugsubreport-routing subreport-ipmgmt

group-SignalingProtocol

Itcollectsthefollowingreportsrelatedtothesignalingprotocolsandthesignalingnetworkmanager:

subreport-signalingSS7 subreport-signalingSCCPsubreport-signalingNetManager subreport-signalingPacketCaptureSctp subreport-signalingPacketCaptureSccpUser subreport-syslog subreport-corefiles subreport-alarmsinfo subreport-debug

TroubleshootingMulticontrollerRNC Symptomsdatacollection

Issue:02B DN0976768 19

Table 3 Supportedplugins(Cont.)

Plugin Name

Description

group-IpalPlatform Thisisthereportgroupcontainingthefollowingsub-reports:subreport-IpalSignalling subreport-IpalCallManagement subreport-IpalCommonsubreport-IpalTransportsubreport-IpalBasicServices

subreport-alarmsinfo Itcollectsalarmdataforaselectedperiodoftime.

subreport-clusterinfo Itcollectsclusterandnodeinformation.

subreport-clusterstatus

Itcollectsinformationaboutthestatusofcluster,nodes,recoverygroupsorunits.

subreport-corefiles Itcollectscoredataavailablewithinthe/crashdirectoryoftheactivenode.

subreport-databaseinfo

ItcollectsthedatabaseinformationsuchasDatabaseDeployed,DiskSpace,PostgresConfigurationfiles,Processlists,andEnterpriseDatabasedetails.

subreport-debug Itcollectsdebugdatacollectedfrommasterdebuglog.

subreport-fastpath Itcollectsinformationfromnodeswhichhave6windipstack.Forexample,ngctl.dat,fpdebug,fpstat.

subreport-ipmgmt ItcollectstheIPnetworkingconfigurationdata.

subreport-licinfo Itcollectslicensemanagementrelatedinformation.

subreport-licdebug Itcollectslicensemanagementrelatedinformation.

subreport-ipsec ItcollectsinformationaboutIKEtemplate,IPsecrules,VPNs,andconfigurationfiles.

subreport-MessageMonitor

Itcollectstwomessagelogsforthefollowingtraffic:

callscenarios operationandmaintenace(O&M)scenarios

Messagemonitoringparametersarestoredinthemsgmon.ini fileunder/opt/nsn/SS_SysReport/stdsymp/plugins/path.

Bydefault,thebufferssizeissetto8MB(0x8FFFFF),anddurationofthemonitoringis30seconds.Themessagemonitoringisperformedintworuns,thefirstisforcallscenarios,andthesecondisforO&Mscenarios.

Tomodifythesettingsthatareusedtocapturemessagelogswithsubreport-MessageMonitor,proceedasfollows:

Symptomsdatacollection TroubleshootingMulticontrollerRNC

20 DN0976768 Issue:02B


Plugin Name

Description

1. Switchto rootuserandbashshell(typeexitlaterontoreturntofsclish).set user username rootProvidethepasswordforroot(defaultisroot).

2. Copythemsgmon.ini fileto/mnt/backup/shareandmodifyit.cp /opt/nsn/SS_SysReport/ stdsymp/plugins/msgmon.ini /mnt/backup/ share/msgmon.ini

Torestoredefaultsettings,deletethemsgmon.ini filefrom/mnt/backup/sharedirectory.

subreport-osinfo Itcollectsinformationaboutoperatingsystemversion,clusteruptime,diskusage,sharedmemory,buildlabel,imagevariants,currentsnapshotname,andcompleteRPMinformation.

subreport-processinfo Itcollectsprocess-relatedinformation.

subreport-routing Itcollectsthecurrentroutinginformation,IPaddress,androutinginstances.

subreport-signalingNetManager

Itcollectsthering-basedbuffertracefilescreatedfornet-managerinthefollowinglocation:

/trace

subreport-signalingPacketCaptureSccpUser

Itcollectsthetcpdumppcapfilescreatedforcapturingtheflowofpacketsbetweensccpandsccp-user.

subreport-signalingPacketCaptureSctp

ItcollectsthetcpdumppcapfilescreatedforcapturingtheflowofpacketsattheSCTPlevel.

subreport-signalingSCCP

ItcollectsthetracefilesforSCCPsubsystem,ldapconfiguration,andoutputofSCLIshowcommandatthestartingandstoppingtimeofsignalingdiagnostics.

subreport-signalingSS7

ItcollectsthetracefilesforSS7subsystem,ldapconfiguration,andoutputofSCLIshowcommandatthestartingandstoppingtimeofsignalingdiagnostics.

subreport-syslog Itcollectsrelevantdatafromsyslog.

subreport-techservice Itcollectstheoutputofsomesoftwarecommandslikefsswcli,unameandsoon,andalsosomescriptsrequiredbytechservicepeople.


Issue:02B DN0976768 21


Plugin Name

Description

subreport-tracemgmt Itcollectstracing-relatedinformationusefulfordebuggingpurposessuchas:

buildversion tracing-relatedrpmversions configurationfiles LDAPfragmentsundertracing featuresrelatedtotracingLDAPfragment contentsoffptl_adminsharedlibrary listoftracefiles listofpluginlibraries processstatesofNodeTraceManager(NTM)and

ClusterTraceManager(CTM) listofbuffers/adminbuffers diskusageoftracing-relatedfilesystems consistencyoftracingconfiguration

subreport-tracesnapshot

Itcollectsthesnapshotfordefault_platformbufferintheCLA-0node.

subreport-IpalBasicServices

Itcollectsdataondistributedcomputingservices,suchasfunctionalunitstates,nameservicesinformation,andfeaturemanagementdata.

subreport-IpalCallManagement

Itcollectsinformationaboutcallmanagementanduserplanemanagement,suchascalldetails,connectioninformation,anduserplaneserviceallocation.

subreport-IpalCommon Itcollectsgeneralinformationfortroubleshooting.Thisincludessoftwarebuildversion,hardwareconfigurationinformation,recoveryunits/recoverygroups/nodeconfiguration,andbasiccounters.

subreport-IpalSignalling

Itcollectsinformationaboutsignalingconnections,includingallNBAP/SCTPlinkinformationandallSIGTRANlinkinformation.

subreport-IpalTransport

ItcollectsinformationabouttraffictransportthatishandledbyEIPU,suchastransportforwardingtable,GTPtunnelIDinformation,IPBRconfiguration.

subreport-rnchw ItcollectsinformationonMulticontrollerRNCfunctionalunits.

subreport-rncinfo Itcollectsinformationonthesoftwarebuilds,diskspaceusage,availablesnapshots,recoverygroupsstates,PRFILEparameters,andsoon.

Thisincludesalsorelevantinformationfromthesyslog.


22 DN0976768 Issue:02B


Plugin Name

Description

subreport-rncipconfig ItcollectsinformationontheIP-relatedconfiguration,thatis,IPinterfaces,IPaddresses,routing,userplaneresources(ipbrandiproobjects),andsoon.

subreport-rncsignaling

ItcollectsinformationonSIGTRAN-relatedconfiguration,thatis,statusoflocalandremotesubsystems,localapplicationserver(s),SCTPassociations,SCTPprofiles,M3UAlimits,SSPfiltertimers,andsoon.

subreport-rncrnw ItcollectsinformationontheRNWstatus,thatisBTSOMconnectionsstatus,NBAPconnectionstatus,WBTSandWCELobjectsstatus,IPNBobjectstatus,IUCSandIUCSIPobjectsstatusesIUPSandIUPSIPobjectstatuses.

subreport-rnchas Itcollectsinformationonthenode,recoverygroups,andrecoveryunits.

ThisincludesalsoHASrelatedinformationfromthesyslog.

subreport-rncalarm Itcollectsinformationaboutactivealarms,alarmshistory,andsoon.

subreport-rncuplane Itcollectsinformationontheuserplaneconfigurationandstatus.

subreport-rncmon ItcollectsinformationoncontrolplaneUEspecificmonitoringforabnormalcallsaspartofHPLloggingfunctionality.

3.2 Collecting standard symptom reportsPurpose

Followthisproceduretosave(collect)thestandardsymptomreports.

Steps

1 Save or collect the standard symptom reports.Tosave(collect)thestandardsymptomreport,enterthefollowingcommand:save symptom-report name Thefullsyntaxofthecommandis:save symptom-report name {[exclude [include ] [quick-mode ] [timeout ] [report-max-size ] [single-file-max-size ]} from date-time to date-time


Issue:02B DN0976768 23

w NOTICE:Itisrecommendedtousenormal-modeforthesymptomsdatacollectionforgroup-RNC.Itisbecause,forgroup-RNC,thedatacollectioninnormal-modecompletesinthetimeallocatedforquick-mode.

Simultaneousmultiplesessionsofsymptomreportsdatacollectionwiththesamereportnameisnotallowed.Ifareportfilealreadyexists,collectingstandardsymptomreportcannotbeinitiatedwiththesamereportname.Whenusingthe"from date-time"and"to date-time"optionswiththestandardsymptomreportgroup-RNC,thesyslog andsyslog.1filesarenotfilteredaccordingtothetimerangeprovided.Thetimerangeisnotapplicabletosyslogandsyslog.1files.Symptomreportsthatareolderthan10daysareremovedfromthefilesystemautomatically.Theparametersinthecurlybracketsarenotallowedinthecommandsyntaxafterthe "from date-time"and"to date-time"parameters.Whenretrievingsymptomreportsfromthepreviousyear,thevalueofthefromdate-time andto date-time parametersdonotgobackfurtherthan333daysfromthecurrentdate.Thefollowingmessagesaredisplayedaccordingtothetriggeringscenariosencounteredduringcommandexecution:

Usecase1:Thedatespecifiedinthefrom date-time parameterisfromthepreviousyear.Messagedisplayed:From/To dates cannot be older than 333 days from current dateDate has to be given in specified format, refer usage

Usecase2:Thedatespecifiedinthefrom date-time parameterisfromthepreviousyearwhilethedatespecifiedintheto date-time parameterisinthecurrentyear.Messagedisplayed:From/To dates cannot be older than 333 days from current dateDate has to be given in specified format, refer usage

Iftheuseronlyspecifiedthefrom date-time parameter,oriftheuseronlyspecifiedtheto date-time parameter,thentheonethatwasnotspecifiediscalculatedbyadefaultoffsetof10days.Forthedescriptionoftheparametersofthesave symptom-report command,seeTable4:Parametersrelatedtosavesymptom-reportcommand.

Table 4 Parametersrelatedtosave symptom-reportcommand

Parameter

Description

name Thismandatoryparametercreatesasingleormultiple(itdependsonsubsequentoptions)tarfile(s)withtheprovidedreport-name.Thereport(s)arestoredunderthedirectory/mnt/backup/stdsympoftheMulticontrollerRNC.Thereportfile(s)generatedcontainsubreport(s)thatarecompressedarchivefile(s)(with.tar.gzextension)containingoneormorefile(s).


24 DN0976768 Issue:02B

Table 4 Parametersrelatedtosave symptom-reportcommand(Cont.)

Parameter

Description

Inthereportfile(s),thereisalsoonetextfilethatsummarizesexecutedcommands,theirexecutionstatusandtime.Itsformatis:__summary.txt

Specialcharactersforreport-namearenotallowed.Numbersarenotallowedasfirstcharacter.Thereportnamelengthmustnotexceed25characters.

[exclude ] Ifthisoptionalparameterisused,thestandardsymptomreportoftheselectedpluginandgrouparenotcollected.

[include ] Ifthisoptionalparameterisused,thestandardsymptomreportcanbecollectedfromaspecifiedpluginorgroup.

[quick-mode ] Thisoptionalparameterisusedinemergenciestogatherimportantandrelevantinformationratherthanhavingallinformationavailable.Whenquickmodeisenabled,thesymptomsubreportsautomaticallyincludedatathatiscriticalandquicklyretrieved.If"yes"isselected,eachsubreportincludesonlydatathatisquicktocollect.If"no"isselected,eachsubreportincludesalldatathatisneededinasupportcase.Thedefaultvalueforthisoptionis"no".

[single-file-max-size]

Thisoptionalparametercreatesthefinalreportinchunksorpiecesoftarfile,eachwithamaximumsizespecifiedastheargument.Eachchunkisatarfilewhichcontainsreportsthatareindependentlyanalyzable.Reportproducedbysinglesubreportisnotsplitevenifthereportsizeexceedsthespecifiedmaximumsize.Thisimpliesthatsizeofasinglefilemaysometimesbebiggerthanthespecifiedmaximumsinglefilesize.Thedefaultvalueforthisoptionis10MB.Asizeof0meansnolimit.

[report-max-size ]

Thisoptionalparameteracceptsthemaximumallowedsizelimit(inMB)forthedatagenerated.Whenthesizeofthedatageneratedreachesthespecifiedlimit,theframeworkstopsthedatacollectionafterthe


Issue:02B DN0976768 25

Table 4 Parametersrelatedtosave symptom-reportcommand(Cont.)

Parameter

Description

currentlyexecutingsubreportcompletesitsexecution.Sincethedatagenerationisnotabruptlystoppeduponreachingthesizelimit,theresultingsizeofthesymptomdatacollectedmaystillexceedthespecifiedlimit.Thedefaultvalueforreport-max-sizeis30MB.Asizeof0meansnolimit.

[timeout ] Thisoptionalparameterspecifiesthetimeoutvalueforsymptomdatacollection.Whentimeoutoccurs,thepluginthatiscurrentlybeingprocessedfailstobecollected,andallthequeuedplugins(ifany)areskipped.Timeoutoptionisstrictlyfollowedexceptwhenpluginsgointokernelmode.Inkernelmode,delaysoccurbecausethesignalsarenotprocesseduntilitlogsoutofthekernelmode.Permittedrangeis0-60.Avalueof0meansthereisnotimeout.Timeoutoptionallowsyoutostopthesymptomdatacollectionafterthespecifiedtimeoutvalueexpires.Thedefaulttimeoutis30minutes.

from date-time

Thisoptionalparametersavessymptomreportuptoaparticulardate.AccepteddateformatsareYYYY.MM.DD-HH:MM:SSorDD.MM.YYYY-HH:MM:SS.Whentryingtocollectsymptomreportsfromthepreviousyear,makesuretheparameterdoesnotgobackfurtherthan333daysfromthecurrentdate.

to date-time Thisoptionalparameterspecifiesthedatetillwhichthestandardsymptomreportmustbecollected.TheaccepteddateformatsareYYYY.MM.DD-HH:MM:SSorDD.MM.YYYY-HH:MM:SS.Whentryingtocollectsymptomreportsfromthepreviousyear,makesuretheparameterdoesnotgobackfurtherthan333daysfromthecurrentdate.

Examples

a) Tocollectthebasicsymptomreportwithaspecificname(RNC311),andwithasubreportgroup(group-RNC),executethefollowingcommand:save symptom-report name RNC311 include group-RNC


26 DN0976768 Issue:02B

b) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),andwithamessagemonitoring(subreport-MessageMonitor),executethefollowingcommand:save symptom-report name RNC311 include group-RNC include subreport-MessageMonitor

c) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),andfromthegivendate(2013.08.05-09:00:00),executethefollowingcommand:save symptom-report name RNC311 include group-RNC from date-time 2013.08.05-09:00:00

d) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),fromthegivendate(2013.08.05-09:00:00)toaparticulardate(2013.08.06-09:00:00),executethefollowingcommand:save symptom-report name RNC311 include group-RNC from date-time 2013.08.05-09:00:00 to date-time 2013.08.06-09:00:00

e) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),fromthetimetheMulticontrollerRNCwascommissionedtoaparticulardate(2013.08.06-09:00:00),executethefollowingcommand:save symptom-report name RNC311 include group-RNC to date-time 2013.08.06-09:00:00

f) Tocollectallthestandardsymptomreportswithaspecificname(RNC311),executethefollowingcommand:save symptom-report name RNC311

g) Tocollectallthestandardsymptomreportswithaspecificname(RNC311ipconifg),andwiththespecificsubreport(rncipconifg),executethefollowingcommand:save symptom-report name RNC311ipconfig include subreport- rncipconfig

h) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),andwiththelimitedsizeofasinglereportfile(5MB),executethefollowingcommand:save symptom-report name RNC311 include group-RNC single-file-max- size 5

i) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),andwiththemaximumsizeofthereport(10MB),executethefollowingcommand:save symptom-report name RNC311 include group-RNC report-max-size 10

j) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),andwithtimeoutfordatacollection(10minutes),executethefollowingcommand:save symptom-report name RNC311 include group-RNC timeout 10

k) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),andwiththesinglesubreportexcludedfromthegroup(subreport-rnchw),executethefollowingcommand:save symptom-report name RNC311 include group-RNC exclude subreport-rnchw

Expected outcome:Thefilesarestoredasindirectory/mnt/backup/stdsymp,withthenameusedinthesave symptom-reportcommand.


Issue:02B DN0976768 27

3.3 Listing the standard symptom reportsPurpose

Followthisproceduretolistthestandardsymptomreportoffiles,pluginsorgroupsavailableontheMulticontrollerRNC.

Before you start

Defaultsystemusernameandpasswordare"_nokadmin"/"system".

Steps

1 List the symptom reports.Tolistallthestandardsymptomreports,enterthefollowingcommand:show symptom-report all

Sample OutputCFPU-0@RNC-311 [2013-08-06 07:52:43 +0200]ReportName FileName Chunk/Total Store---------- -------- -------- -----AllRNC-311 /stdsymp/AllRNC-311_0.tar part 1/2 CFPU-0AllRNC-311 /stdsymp/AllRNC-311_1.tar part 2/2 CFPU-0MessageMonitor /stdsymp/MessageMonitor_0.tar part 1/1 CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 CFPU-0rncmon /stdsymp/rncmon_0.tar part 1/1 CFPU-0RoutingFails /stdsymp/RoutingFails_0.tar part 1/1 CFPU-0history /stdsymp/history_0.tar part 1/1 CFPU-0

w NOTICE:TheChunk/Totalfielddisplaysthenumberofchunkspresentfortheparticularreport.Thisassiststheuserinlearningwhetherthefilesofthereporthavebeenlistedproperly.

2 List the detailed contents of a particular symptom report.show symptom-report name

ExampleTolisttheparticularstandardsymptomreport(RNC-311),enterthefollowingcommand:show symptom-report name RNC-311


28 DN0976768 Issue:02B

Sample OutputCFPU-0@RNC-311 [2013-08-06 07:57:14 +0200]ReportName FileName Chunk/Total Subreport Store---------- --------- --------- ----- -----RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rnchw CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rncinfo CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rncipconfig CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rncsignaling CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rncrnw CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rnchas CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rncalarm CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rncuplane CFPU-0

3 Display the list of available plugins.ExampleTodisplaythelistofavailableplugins,enterthefollowingcommand:show symptom-report plugin-list

4 List the group of plugins based on the group names.ExampleTolistthegroupofpluginsbasedonthegroupnames(group-RNC),enterthefollowingcommand:show symptom-report group group-RNC

3.4 Copying standard symptoms reports to remotemachinePurpose

FollowthisproceduretocopystandardsymptomsreportsfromMulticontrollerRNClocalstoragetoremotemachine.

Before you start


Expected outcome


Issue:02B DN0976768 29

Thestandardsymptomsreportsaresuccessfullycopiedtoexternalstorage.

Steps

1 Switch to bash shell (type exit later on to return to fsclish).shell bash full

Confirmthecommandbypressing"y".

2 Use SCP to copy standard symptoms report to remote machine.Notethatthereportfilesarestoredin /mnt/backup/stdsympdirectory.scp /mnt/backup/stdsymp/.tar @:/Notethatyoucanusewildcardsymbol"*"inthetocopyallthereportfilesfrom/mnt/backup/stdsympdirectory.IfthesameremotemachineisusedtostorethereportsfromdifferentMulticontrollerRNCs,itisrecommendedtoincludespecificMulticontrollerRNCnameandidentifierinthefoldernametodifferentiatethereports.

w NOTICE:ThereportfilescanalsobecopieddirectlyfromtheMulticontrollerRNCtoexternalmachineusingSFTPorSCPprotocol.Thereisadefaultaccount_nokadminwhichprovidestheaccesstoMulticontrollerRNCfilesystem.Thedefaultpasswordfor_nokadminuseris"system.Thereportsarestoredunder/stdsympdirectory.

3.5 Deleting the standard symptom reportsPurpose

Followthisproceduretodeletethestandardsymptomreports.

Before you start


Steps

1 Delete the particular standard symptom report.Todeleteaparticularstandardsymptomreport,enterthefollowingcommand:delete symptom-report name

ExampleTodeleteaparticularstandardsymptomreportwithnameRNC311,enterthefollowingcommand:delete symptom-report name RNC311

Sample Output:CFPU-0@RNC-311 [2013-08-06 08:29:10 +0200]ReportName FileName Store


30 DN0976768 Issue:02B

---------- -------- -----RNC311 /stdsymp/RNC311_0.tar CFPU-0

2 Delete all the standard symptoms reports.Todeleteallthestandardsymptomreports,executethefollowingcommand:delete symptom-report all

Sample Output:CFPU-0@RNC-311 [2013-08-06 08:32:00 +0200]ReportName FileName Store---------- -------- -----AllRNC311 /stdsymp/AllRNC311_0.tar CFPU-0AllRNC311 /stdsymp/AllRNC311_1.tar CFPU-0MessageMonitor /stdsymp/MessageMonitor_0.tar CFPU- 0RoutingFails /stdsymp/RoutingFails_0.tar CFPU- 0history /stdsymp/history_0.tar CFPU- 0rncmon /stdsymp/rncmon_0.tar CFPU- 0


Issue:02B DN0976768 31

4 Generic software troubleshootinginstructions

4.1 Displaying all blackbox filesPurpose

Thissectionprovidesinstructionsonhowtodisplayalltheblackboxfilesinthesystem.Theseblackboxfilesarenormallylocatedin/srv/Log/crashfolder.

Before you start

Makesureyouhavetheauthoritytothesecondarygroup_nokfsuicrashlogandthepermissionfsASView.Toavoidaddingtheuseraccounttoalonglistofgroups,youarerecommendedtoaddthepermissionfsASViewtogroup_nokfsuicrashlogusingthecommandadd user-management group-to-permission gid_nokfsuicrashlog permid fsASViewandthenassignthegroup_nokfsuicrashlogtothetargetuseraccount.

1 Show all blackbox file groups for all crashed processes.Enterthefollowingcommand:show troubleshooting blackbox listExpected outcomeAlltheblackboxnamesarelisted.Eachnameindicatesthereisaprocesscrash.Anexampleoftheoutputisasfollows.Apr 10 16:34 CLA-1-11945-534657b0-snmpmdserver-ABRTApr 21 09:27 CLA-0-11435-53547401-snmpmdserver-ABRT

Steps

1 Show all blackbox file groups for all crashed processes.Enterthefollowingcommand:show troubleshooting blackbox listExpected outcomeAlltheblackboxnamesarelisted.Eachnameindicatesthereisaprocesscrash.Anexampleoftheoutputisasfollows.Jun 23 14:34 CFPU-0-21089-4e02db77-sokeri-ABRTJun 23 15:30 CFPU-0-21441-4e02eb8e-ilalarm-ABRTJun 24 14:24 CFPU-0-7052-4e042aaa-slapd-ABRTJul 18 13:15 CFPU-0-7905-4e23c149-lastproc-ABRTJul 18 13:15 CFPU-0-7871-4e23c188-starter-ABRT

Genericsoftwaretroubleshootinginstructions TroubleshootingMulticontrollerRNC

32 DN0976768 Issue:02B

4.2 Viewing active alarms and alarms historyThealarmsystemindicatespotentialfaultsinthesystemaswellasfaultsthatrequirecorrectiveactions.Afteranalarmisraised,thefaultcausingthealarmmustbesolved.Thesolutioncanbeanautomaticrecoveryoramanualcorrectiveaction.Alarmsaretypicallyusedinsituationswhereitispossibletogiveinstructionsforcorrectiveactionsinthealarmdescription,suchasreplacingahardwareunit.

Thealarmscanalsoberaisedthroughthealarmsystemtoindicatethatthesystemisnotworkingnormally,forexample,whentheharddiskisfullandthesystemcannotwritetoit.Suchalarmsareclearedautomaticallywhenthesystemreturnstoitsnormalstate.Youcannotclearthesealarmsmanually.

Multicontroller RNC alarm handling MulticontrollerRNCOMSprovidesFaultManagementapplicationthatenablesuser

toperformoperationssuchas:

managingMulticontrollerRNC,MulticontrollerRNCOMS,andFlexiBTSalarms viewingMulticontrollerRNC,MulticontrollerRNCOMS,andFlexiBTSalarms

history

Forinformationonhowtocheckandmanagethealarms,seeManaging Faults withOMS.

g TochecktheactivealarmsandalarmshistorydirectlyfromMulticontrollerRNCcommandlineinterface,enterthefollowingcommands:# zahoUsage: zaho [ -h | --help | -a | --alarms | -p | --alarmhistory ][ -x | --alarmx | -s | --summary ] Obligatory parameters: -h | --help : this help -a | -- alarms : active alarms -x | --alarmx : shows alarms with certain alarm number -s | --summary : summary of the alarmsExample: zaho -a : prints all active alarms zaho -ax 12345 : prints all 12345 alarms from active alarms zaho -ps : prints summary about alarm history

ForinformationontheMulticontrollerRNCspecificalarms,checkthefollowingreferencedocumentation:Multicontroller RNC Notices (0-999)Multicontroller RNC Disturbances (1000-1999)Multicontroller RNC Failure Printouts (2000-3999)Multicontroller RNC, IPA-RNC and I-HSPA Adapter Base Station Alarms (7000-7900)

Flexi BTS alarm handling FlexiBTSalarmsaremanagedusingOMSFaultManagementapplication.Asan

alternative,theycanbealsocheckedbyusingBTSSiteManager.Formoreinformation,seeChecking BTS alarmsofTroubleshooting Flexi Multiradio BTSWCDMA.

ForinformationontheFlexiBTSspecificalarms,see Flexi Multiradio BTS WCDMAFaults.

TroubleshootingMulticontrollerRNC Genericsoftwaretroubleshootinginstructions

Issue:02B DN0976768 33

Multicontroller RNC OMS alarm handling MulticontrollerRNCOMSalarmsaremanagedusingOMSFaultManagement

application. ForinformationonthespecificMulticontrollerRNCOMSalarms,see:OMS Alarms.

4.3 Collecting core dump file informationPurpose

Whenanapplicationcrashes,itwritesthecontentsofitsexecutionenvironment(stacks,variablesandsoon)intoafile.Thisfileisknownasacoredump.

Acoredumpisusefulforproblemsolving,sinceitprovidesthefollowingdetails:

Theapplication'staskduringfailure Theapplication'sexecutionenvironment

Inadditiontothecoredump,thesystemalsogathersinformationsuchasrecentsyslogentriesandinformationaboutthenode.Thisinformationisalsosavedtofilesandlabeledthesamewayasthecorefiles.

Before you start


Steps

1 Open an SSH connection to Multicontroller RNC IP address.

2 List all the blackbox files in the system.show troubleshooting blackbox listTheseblackboxfilesarenormallylocatedin/srv/Log/crashfolder.

3 Switch to root user and the bash shell.set user username root

Providethepasswordforrootuser(defaultisroot).

4 Collect the log and the core files.Tocheckthefilesthatcanbeaccessedfromthenode,wherethelogrecoverygroupisrunning,enterthefollowingcommand:ls /srv/Log/crashThefilenameswillbeinthefollowingformat:----..gzWhere:

nodeisthenameofnodewherethecrashoccurred.

Genericsoftwaretroubleshootinginstructions TroubleshootingMulticontrollerRNC

34 DN0976768 Issue:02B

pidistheprocessidofthecrashedprocess. timestampisthetimeofcrashinsecondssince1970-01-0100:00:00UTCin

hexadecimalnotation. nameisthenameofthecrashedprocess. signalisthesignalthatcausedtheprocesstodumpthecore.

typecanbeoneofthefollowing:

core-thecoredumpfile syslog-asnapshotofsyslogentriesfromthetimeofcrash debug -asnapshotofdebugentriesfromthetimeofcrash blackbox.tar -informationaboutthenode

5 Copy the files from the Multicontroller RNC to external machine.scp /srv/Log/crash/ @://

6 Send the files to your Nokia Solutions and Networks representative.

TroubleshootingMulticontrollerRNC Genericsoftwaretroubleshootinginstructions

Issue:02B DN0976768 35

5 Configuration management troubleshooting

5.1 Saving configuration snapshot fails5.1.1 Description

Ifthereisnofreespaceonthedisk,savingtheconfigurationsnapshotisnotpossible.

5.1.2 SymptomsThesave snapshotcommandfailswithanerrormessage.

5.1.3 Recovery proceduresIfsavingconfigurationsnapshotfails,followeitherofthesesteps:

1 Check the existing configuration snapshots and delete the old configurationsnapshots that are not needed.Todisplaytheexistingconfigurationsnapshots,enterthefollowingcommand:show snapshot listTodeleteallconfigurationsnapshotsthatarenolongerneeded,enterthefollowingcommand:delete snapshot config-name

2 Check the existing software volumes and delete the old software deliveriesthat are not needed.Todisplaytheexistingsoftwarevolumes,enterthefollowingcommand:show sw-manage listTodeletetheoldsoftwaredeliveriesthatarenolongerneeded,enterthefollowingcommand:delete sw-manage delivery

Configurationmanagementtroubleshooting TroubleshootingMulticontrollerRNC

36 DN0976768 Issue:02B

6 Software management troubleshooting

6.1 Installation of a new software delivery fails6.1.1 Description

Theinstallationofanewsoftwaredeliveryfailsifthesoftwareimageiscorruptedoralreadyinstalled.

6.1.2 SymptomsItisnotpossibletoinstallanewsoftwaredelivery.

6.1.3 Recovery proceduresDefaultsystemusernameandpasswordare"_nokadmin"/"system".

1 Check if the software image is corrupted.Ifthesoftwareimageiscorrupted,copyitagainfromtheprovidedtransfermedium.

w NOTICE:Theintegrityofthesoftwaredeliveryimagecanbeverifiedbyusingthemd5sumtool.

2 Check if the image to be installed is already existing.Iftheimagetobeinstalledalreadyexisting,thenthenewsoftwareimagecannotbeinstalledonthesystem.

3 Check the log files related to software management.

a) Switchtobashshell(typeexitlaterontoreturntofsclish).Confirmthecommandbypressing"y".

b) Logfilesarestoredtothefollowinglocations:/var/log/ilsw_upgrade.log/var/log/swm.log

6.2 Activation of a new software delivery fails6.2.1 Description

InthemcRNChardware,whenactivatinganewsoftwaredelivery,thereisafallbackmechanismtoactivatethepreviousdeliveryifthenewdeliverydoesnotbootup.Thesystemswitchestothepreviousdeliveryafterfivefailedbootattempts.

TroubleshootingMulticontrollerRNC Softwaremanagementtroubleshooting

Issue:02B DN0976768 37

6.2.2 SymptomsAfteractivationofanewsoftwaredelivery,thesystemtakesalongtime(aboutfivetimeslongerthannormally)tobootup.Whenthesystemisoperationalagain,thepreviousdeliveryisactivatedinsteadofthenewsoftwaredelivery.

6.2.3 Recovery proceduresDefaultsystemusernameandpasswordare"_nokadmin"/"system".

1 Contact Nokia Solutions and Networks representative and attach the followinglog files to carry out the recovery actions.

a) Switchtobashshell(typeexitlaterontoreturntofsclish).Confirmthecommandbypressing"y".

b) Logfilesarestoredtothefollowinglocations:/var/log/ilsw_upgrade.log/var/log/swm.log/var/log/fsconfigure.log

6.3 Troubleshooting software configurationmanagement

6.3.1 Executing pre-download script failsSymptomsTheexecutionofDownloadtaskonNetActinterfacefails.Thefollowingerrorlogs(asanexample)arefoundinthe ilsw_upgrade.logfile(thatlocatesin/var/logfolderonmcRNC):

2011-09-14 17:56:58 ILSWMAN: Execute pre_downloading_02_fail.sh error! Status: 12011-09-14 17:56:58 SOKERI: Download pre-checking error! Status: 255, Additional Status: 25

Reason analysisBeforedownloadingthesoftwaredelivery,thesystemexecutesthepre_downloading_xx.shscripttodopre-downloadinspectionifsuchscriptexists.AnerrorintheexecutionofthescriptcausesthefailureoftheDownloadtask.

Recovery proceduresSteps

1 Make a copy of syslog file and ilsw_upgrade.log file (both under /var/logdirectory) respectively.

Softwaremanagementtroubleshooting TroubleshootingMulticontrollerRNC

38 DN0976768 Issue:02B

2 Send the copied files to Nokia Customer Service Center.

6.3.2 Parsing targetBD.xml failsSymptomsTheexecutionofDownloadtaskonNetActfails.Thefollowingerrorlogs(asanexample)arefoundinthe ilsw_upgrade.logfile(thatlocatesin/var/logfolderonmcRNC):

2011-09-14 16:35:58 SOKERI: Parse target BD SWPackages/default/TargetBD.xml start.2011-09-14 16:35:59 SOKERI: Parse target BD error! Status: 17466, Additional Status: 18

Reason analysisWhentheoperatorstartsaDownloadtaskonNetAct,mcRNCdownloads targetBD.xmlfilebeforedownloadingtheactualsoftwaredelivery.ThereasonforthefailedparsingofthetargetBD.xmlfilecanbe:

TheformatofthetargetBD.xmlfileisincorrect. Keyinformationismissinginthefile.

g Inthisexample,thetargetBDfileisnamedastargetBD.xml.Thisnamecouldbedifferentintherealenvironment.

Recovery procedures

1 Make a copy of the needed filesThefilesthatneedtobecopiedare:

targetBD.xmllocatedin/var/opt/nsn/SS_ILOMU/SS_ILSWMAN/build/ folder sysloglocatedin/var/logfolder ilsw_upgrade.loglocatedin/var/logfolder


6.3.3 Executing pre-activation script failsSymptomsTheexecutionofDownload and ActivateorActivatetaskonNetActfails.Thefollowingerrorlogs(asanexample)arefoundintheilsw_upgrade.logfile(thatlocatesin/var/logfolderonmcRNC):

2011-09-16 09:35:56 ILSWMAN: Execute pre_activation_02.sh start2011-09-16 09:35:56 ILSWMAN: Execute pre_activation_02.sh error! Status: 2552011-09-16 09:35:57 SOKERI: Pre-Activate error! Status: 255, Additional Status: 25


Issue:02B DN0976768 39

Reason analysisBeforeactivatingthesoftwaredelivery,thesystemexecutesthepre_activation_xx.shscripttodopre-activatinginspectionifsuchscriptexists.AnerrorintheexecutionofthescriptcausesthefailureoftheDownload and ActivateorActivate task.Thereasonforthefailedexecutionofthescriptvariesdependingonwhatarespecifiedinthescript.

Recovery procedures

1 Make a copy of syslog file and ilsw_upgrade.log file (both under /var/logdirectory) respectively.


6.3.4 Upgrading eSW failsSymptomsTheeSWupgradefailsduringthemcRNCupgrade.ThestatusINSTALLATION_FAILEDisdisplayedontheACTIVE/MAINBANKorthePENDINGBANKfieldwhenyouexecutetheSCLIcommandshow sw-managed embedded-sw status.

Reason analysisAsuccessfuleSWupgradehascertainrequirementsonthediskspace.Whenthefreediskspaceinthe/tmpfolderisbelow7MB,ortheusageofthe/dev/mtdblock1filesystemexceeds83%,theeSWupgradefails.

Recovery procedures

1 Enter the bash shell as a root user.Toenterthebashshellasarootuser,executethefollowingcommand:shell bash full

2 Check the available space of each LMP.TochecktheavailablespaceofLMP,executethefollowingcommandforeachLMP:ssh LMP-1-X-1 "df h"

Example output for LMP-1-1-1# ssh LMP-1-1-1 "df -h" Filesystem Size Used Avail Use% Mounted on /dev/mtdblock1 117M 103M 14M 88% / tmpfs 506M 2.6M 504M 1% /dev/shm /dev/mtdblock0 10M 2.6M 7.5M 26% /boot none 32M 31M 1M 97% /tmp none 64M 0 64M 0% /mnt/var none 64M 22M 43M 34% /opt/fastpath/tmpfs none 64M 22M 43M 34% /opt/fastpath2/tmpfs /dev/mtdblock5 117M 96M 21M 83% /mnt/backupfs


40 DN0976768 Issue:02B

3 Delete some unnecessary files to free more space.Deletesomeunnecessaryfilesinthe/tmpdirectoryorroot/filesystemofeachLMPuntil:

Thefreediskspaceinthe/tmpdirectoryisabove7MB. Theusageofthefilesystemmountedonroot/(/dev/mtdblock1intheoutput

ofstep2)isbelow83%.

g Donotdeleteanyfilesthatareimportantforthesystemfunctionality.Forexample,itissafetodeletethefilesinthe/tmp/sysrepdirectory.

6.3.5 Alarm 2518 - NO VALID FALLBACK COPY FOR DEFAULTPACKAGE is triggeredSymptomsAlarm 2518isseenbothfromtheNetActAlarmMonitorgraphicaluserinterface(GUI),andthemcRNCcommandlineinterfacewhenexecutingshow alarm activeSCLIcommand.Anexampleofthecommandoutputisasfollows:

Alarm ID : 550Specific problem : 2518 - NO VALID FALLBACK COPY FOR DEFAULT PACKAGEManaged object : fshaRecoveryUnitName=OMUTestServer- 0,fsipHostName=CLA-0,fsFragmentId=Nodes,fsFra gmentId=HA,fsClusterId=ClusterRootSeverity : 3 (major)Cleared : noClearing : automaticAcknowledged : noAck. user ID : N/AAck. time : N/AAlarm time : 2012-01-04 11:17:47:633 CSTEvent type : x2 (processing error)Application : fshaProcessInstanceName=IL_Sokeri,fshaRecoveryUnitName=OMUTestServer-0,fsipHostN ame=CLA- 0,fsFragmentId=Nodes,fsFragmentId=HA,fsClusterId=ClusterRootIAppl Addl. Info : 4Appl. Addl. Info :

Reason analysisThefaultiscausedbyoutdatedfallback(FB)build.ItisrequiredthattheFBbuildmustbesavedatafrequencyofeverysevendays.

Recovery proceduresSteps

1 Enter SCLI command interface on mcRNCTogetintoSCLIcommandinterface,enterthefollowingcommand:fsclish


Issue:02B DN0976768 41

2 Create an FB buildEnterthecommandasfollows:save sw-manage app-sw-mgmt fb-buildExpected outcomeThefollowingoutputisdisplayed:FB build has been saved successfully.Unexpected outcomeThefollowingoutputisdisplayedwhentheactivebuildisnotabackup(BU)build.Current build is not BU.FormoreinformationoncreatinganFBbuild,seeCreatingfallbackbuildfromactivebackupbuild.

3 Check if Alarm 2518 has been cancelledAfterthenewFBbuildiscreated,thealarmiscancelledautomatically.Youcanseetheresultbyeitherofthefollowingways:

Usetheshow alarm activeSCLIcommandortheshow alarm activefilter-by specific-problem 2518command.Alarm2518stillexistsintheoutput,buttheClearedfieldisnowyesandSeverityhaschangedto6(cleared).Forexample:Alarm ID : 550Specific problem : 2518 - NO VALID FALLBACK COPY FOR DEFAULT PACKAGEManaged object : fshaRecoveryUnitName=OMUTestServer- 0,fsipHostName=CLA-0,fsFragmentId=Nodes,fsFra gmentId=HA,fsClusterId=ClusterRootSeverity : 6 (cleared)Cleared : yesClearing : automaticAcknowledged : noAck. user ID : N/AAck. time : N/AAlarm time : 2011-12-19 16:21:00:749 CSTEvent type : x2 (processing error)Application : fshaProcessInstanceName=IL_Sokeri,fshaRecoveryUnitName=OMUTestServer-0,fsipHostN ame=CLA- 0,fsFragmentId=Nodes,fsFragmentId=HA,fsClusterId=ClusterRootIAppl Addl. Info : 4Appl. Addl. Info :

ChecktheactivealarmsintheAlarmMonitoroftheNetActGUI.Alarm2518hasdisappeared.


42 DN0976768 Issue:02B

7 Networking troubleshooting

7.1 Packet loss in certain traffic flows unexpectedlyhigh

7.1.1 DescriptionThenetworkelementkeepsthepacketdropfromdifferenttrafficflowsundercontrolduetotheQualityofService(QoS)systemfeatureimplementedontheingress(incoming)andegress(outgoing)sides.TheQoSoningresssidemanagescaseswhenNPUisoverloadedandQoSonegresssidemanagescaseswhenoutgoinginterfacecapacityisoverloaded.Inaddition,theinternaltrafficismanaged.

Thepacketlossincertaintrafficflowmaybeunexpectedlyhighbecause:

Theegressport rate-limitthresholdislessthantheconvergedtrafficflowsvolumeofferedandACLrulestosettheproperpipmarkforanyoutgoingstreamsmightnotbecreated

IncomingpacketsmighthavewrongDSCPmark.

7.1.2 SymptomsThepacketlossincertaintrafficflowsunexpectedlyhigh.

7.1.3 Recovery procedures

1 Analyze the traffic flows coming out of the interface.Toverifywhetherthepacketlosshappensintheegressinterface,collecttheegressinterfaceQOSstatisticsbyusingtheperformancemanagementmeasurementjobs,whilethetrafficisongoing.Whileanalyzingthereportfile,lookforanydiscardedpacketcountforthespecificegressinterface.Iftherearenosignificantdiscardedpackets,thenskipstep1andstep2whichhandlesegresssideandproceedtostep3whichhandlestheingressside.

2 Check the rate-limit value for the egress interface.Tochecktherate-limitvaluefortheegressinterface,enterthefollowingcommand:show networking ether owner iface\

Checktherate-limitvalueintheoutput.Ifthe rate-limitvalueisequaltothetrafficpassed,thelowestprioritytrafficmaybedropped.Topreventpacketloss,therate-limitthresholdmaybeincreased.Ifthisactionisnotdesired,thenproceedtostep2.Tosetthenewrate-limit threshold,enterthefollowingcommand:set networking ether iface rate-limit

TroubleshootingMulticontrollerRNC Networkingtroubleshooting

Issue:02B DN0976768 43

g Iftheegressinterfaceisalinkaggregationgroupinterface,thenmakesurethattheratelimitvaluedoesnotexceedthesummationofalltheportsthatwasunderthelinkaggregationgroupinterface.

3 Check if the traffic flow is appropriately handled by egress interface QoS.EachqueuefromaqueuesetattachedtotheinterfaceimplementparticularPHB.ThequeueisselectedforeachpacketaccordingtothePacketInternalPriority.PIPismarkedbyclassificationfunctionoftheACL.Tocheckifthepip-markfortheflowunderconsiderationisconfiguredproperly,enterthefollowingcommand:show networking aclruleThefollowingtableprovidesthemappingbetweenthequeue IDand pip-mark.

Table 5 Mappingbetweenpip-markandQueueID

pip-mark

queue ID

0x00 0

0x01 1

0x02 2

0x03 3

0x04 4

0x05 5

0x06 5

0x07 5

Tomaptheselectedtraffictotherightqueueaslistedinthetable,seethesection,Configuring ACL settings.Thepriorityofthequeueisdeterminedbytheweightassignedtothequeueintheattachedqueueset.Increasingthelengthofthequeuecanalsoreducethepacketlossduringshorttrafficbursts.However,systemresourcesandtrafficlatencyhastobetakenintoaccount.FormoreinformationrefertoConfiguring queue sets section.Also,theothertrafficflow(s)maygethigherprioritythantheflowunderconsideration,thusoccupyingthesystemresources.Tocheckwhethertheattachedqueuesethasappropriateweightandlength,enterthefollowingcommand:show networking qset name Tomodifytheweightandlengthofthequeue,enterthefollowingcommand:set networking qset test1 queue weight \length queue weight \ length

Networkingtroubleshooting TroubleshootingMulticontrollerRNC

44 DN0976768 Issue:02B

4 Verify that the incoming packet has expected DSCP value.

a) ChecktheDSCPvaluefromtheMulticontrollerRNC.

StarttcpdumponMulticontrollerRNCsidetomonitortrafficonthedesiredinterface.

Switchtorootuserandthebashshell.(Typeexitlaterontoreturntofsclish.)set user username rootProvidethepasswordforrootuser.

Enabletcpdumpforselectedinterface.#tcpdump -i -w/mnt/backup/tcp_dump.pcap

OpenthecollectedtracewithprotocolanalyzerandverifytheDSCPvalueofthedesiredtrafficflow.

Or

a) ChecktheDSCPvaluefromtheexternalworkstation.

Runprotocolanalyzerandcapturethetrafficofthedesiredinterface. VerifytheDSCPvalueofthecollectedtrafficflow.

ThefollowingtableliststheexpectedDSCPcodesforexternaltraffic.TheDSCPcodesnotmentionedinthetableareclassifiedasbesteffortclassandarehandledwithlowestpriority.

Table 6 PriorityvaluesfordifferentDSCPcodes

PHB Class

DSCP Code

Priority

nc 0x30 Highpriority

ef 0x2E

af41 0x22 Mediumpriority

af42 0x24

af43 0x26

af31 0x1A Lowpriority

af32 0x1C

af33 0x1E

af21 0x12

af22 0x14

af23 0x16


Issue:02B DN0976768 45

Table 6 PriorityvaluesfordifferentDSCPcodes(Cont.)

PHB Class

DSCP Code

Priority

af11 0x0A

af12 0x0C

af13 0x0E

be 0x00

7.2 Multihop BFD sessions are not established7.2.1 Description

ThemultihopBFDsessionsmaynotbeestablishedifthereareproblemsinBFDservice,configuration,ornetworkconnectivity.

7.2.2 SymptomsThemultihopBFDsessionsarenotestablishedorsessionsdonotreachUPstate.


1 Check the BFD service.TochecktheBFDservice,followthesesteps:

a) VerifythestateoftheFSBFDServerrecoveryunitinthenodewhereBFDisconfigured.Enterthefollowingcommand:show has state managed-object //FSBFDserverThefollowingoutputisdisplayed:root@CFPU-0 [RNC-89] > show has state managed-object /CFPU- 0/FSBFDserverOBJECT ADMINISTRATIVE OPERATIONAL USAGE ROLE PROCEDURAL DYNAMIC/CFPU-0/FSBFDServer UNLOCKED ENABLED ACTIVE ACTIVE - -Iftherecoveryunitislocked,unlockitusingthefollowingcommand:set has unlock managed-object //FSBFDServer

2 Check BFD alarms.VerifythatthereareactivealarmsforBFD(specificproblem70348).Enterthefollowingcommand: show alarm active filter-by specific-problem70348

IftheactiveBFDalarmsarefound,verifyBFDconfiguration(seestep4)andnetworkconnectivity(seestep5).


46 DN0976768 Issue:02B

3 Check the BFD runtime information.VerifythattheBFDsessionsarefoundintheruntimeenvironmentandthatthesessionstateiscorrect.Enterthefollowingcommand:show networking instance bfd runtime node /The Initstatemeansthattheremotesystemiscommunicatingandthelocalsystembringsthesessionup,buttheremotesystemdoesnotrealizeit.Ifthesessionsarenotfound,verifytheBFDservicestateprovidedbyHAS.Ifthesessionstateisdown,verifythebasicIPconnectivity.

g Diagnosticfieldshowsadiagnosticcodespecifyingthereasonforchangeinsessionstate.

4 Check BFD configuration.TochecktheBFDconfiguration,followthesesteps:

a) VerifythatBFDisenabledonthecorrectnodewithcorrectparameters.Enterthefollowingcommands:show networking monitoring bfd config allIftheBFDisnotenableonthecorrectnodewithcorrectparameter,seethesection,Configuring IP Connection for Multicontroller RNCformoreinformation.

b) VerifythatlocalIPaddressforBFDsessionisfoundfromthenodeandisassignedtothecorrectVRFinstance.Enterthefollowingcommand:show networking address

g BFDsessionscanbeconfiguredbeforeIPaddressisaddedtotheinterface.However,theBFDsessioncontrolpacketaresentonlyaftertheIPaddressisadded.IfthelocalIPaddressfortheBFDsessionisnotfoundandassignedtothecorrectVRFinstance,seethesection,ConfiguringtheBFDmultihopsessionsectionformoreinformation.

c) VerifythattherearenoACLrulesthatblockBFDpackets.Enterthefollowingcommand:show networking aclruleIfrequiredaddruleforallowingBFDpacketsinboththedirections,seethesection,Configuring IP Connection for Multicontroller RNCformoreinformation.

g MultihopBFDcontrolpacketsaretransmittedinUDPwithdestinationport4784.

5 Check the IP connectivity between routers.TochecktheIPconnectivitybetweenrouters,followthesesteps:

a) VerifybasicIPconnectivitybetweennetworkdevicesusingpingcommand.PingfromthecorrectVRFinnodeyouhaveconfiguredBFDsession.Ifthereisnoresponsetopingverifytheinterfacestate.ToexecutethepingcommandfromtheSCLI,startthebashshell(infullmode)andthenusesshtoconnecttothecorrectnode.Enterthefollowingcommands:shell bash full ssh ping -V or ping exit


Issue:02B DN0976768 47

IfyouwanttomakesurethatthenodewhereBFDisconfigured,acceptsICMP(protocol1)packetsininputandoutputchain,enterthefollowingcommand:show networking aclrule owner /IfthenodedoesnotacceptICMP(protocol1)packets,thenaddacceptrulesbyenteringthefollowingcommands:add networking aclrule / index 20000chain\ input protocol 1 accept add networking aclrule/ index 20001 chain\ output protocol 1 accept

g Theneighboringroutermaybeconfiguredtodroppingrequestsandmaycausepingnottorespond.b) VerifythatthestatefortheinterfacewhereBFDsessionisconfiguredhasadmin

stateupandhasrunningflagset.Enterthefollowingcommand:show networking interface runtimeIfrunningflagisnotpresent,verifythatcableisproperlyconnectedtotheinterface.Iftheadminstateisnotup,seeConfiguring IP Connection forMulticontroller RNCformoreinformation.

c) VerifythattheBFDcontrolpacketsaresentandreceivedontheinterfacewhereBFDsessionisconfigured.ToexecutethiscommandfromSCLI,startthebashshell(infullmode)andthenusesshtoconnecttothecorrectnode.Enterthefollowingcommands:shell bash full ssh tcpdump -i port 4784 exitThiscommanddoesnotrequire rootaccess.IfonlyoutgoingBFDpacketsareseen,verifytheconfigurationattheremoteendpoint.

w NOTICE: Nokiaadvisesthatonlyadvancedusers,deeplyfamiliarwiththebehaviorofthenetworkelement,mustusecommandsofthefull bashingeneralandthetcpdumpcommandinparticular.Eveniftherootaccountordedicatednonrootaccountaccessisused,thecommandsfromthefull bashshellcanbeeasilyexploited,Forexample,tochangesecurity-relevantconfiguration.Alsothetcpdumpcommandinparticularwhenaccidentallyoron-purposemisusedcancausesignificantperformancepenalties(adropof0-100%)innetworking(throughputandlatencies),diskI/Oorprocessingcapacity.Intheworstcasethismight,forexample,causeautomaticrecoveryactions.Alsosomepartsofthetcpdumpcommandmightnotworkasexpectedduetothepeculiaritiesofthenetworkelementarchitecture.

7.3 OSPF is not working properly7.3.1 Description

IfthereareproblemswithOSPFconfigurations,interfacestate,ornetworkconnectivitytheOSPFneighboradjacenciesarenotformedandthereforeOSPFrouteadvertisementdoesnotwork.

7.3.2 SymptomsOSPFadjacenciesarenotformedandtheroutingtabledoesnotincludeOSPFroutes.ThereisnonetworkconnectivitytonetworksadvertisedbyOSPF.TheOSPFneighborsarenotseenorOSPF not configurederrormessageisdisplayedwhenmonitoringtheOSPFneighboradjacencystatefromtheSCLIcommand.


48 DN0976768 Issue:02B


1 Check the routing service.CheckthestateofFSRoutingDaemonServer recoveryunitinthenodewhereOSPFisconfigured.Enterthefollowingcommand:show has state managed-object //FSRoutingDaemonServerThefollowingoutputisdisplayed:OBJECT ADMINISTRATIVE OPERATIONAL USAGE ROLE PROCEDURAL DYNAMIC//FSRoutingDaemonServer UNLOCKED ENABLED ACTIVE ACTIVE - -IftheFSRoutingDaemonServerrecoveryunitislocked,theOSPFneighborquerytothecorrespondingnodewillresultinanOSPF not configured error.IfFSRoutingDaemonServerrecoveryunitislocked,unlockitusingthefollowingcommand: set has unlock managed-object//FSRoutingDaemonServer

2 Check the OSPF runtime information.TochecktheOSPFruntimeinformation,followthesesteps:

a) Verifythattheneighboradjacencystatehasreached2-wayorfullstate.Enterthefollowingcommand:show networking instance node ospf neighbors

g Thefulladjacencyisformedonlywithdesignatedrouterandbackupdesignatedrouter.Thetimetoreach2-wayorfullstatedependsontheconfiguredOSPFparameters.IfthecommandfailswiththeOSPF not configurederror,verifyroutingservicestate(seestep1)andOSPFconfiguration(seestep3).IfOSPFneighboradjacencyarenotformed,checktheconfiguration(seestep3).

b) VerifythatthebasicIPconnectivitybetweenroutersisworking(seestep4).c) VerifythattheOSPFhellopacketsaretransmittedandreceived.Enterthe

followingcommand:show routing instance node \ospf packetsIftheOSPFhellopacketsarenottransmittedandreceived,checkthatOSPFisconfiguredproperly(seestep3).

d) VerifythatOSPFroutesarefoundinroutingtable.Enterthefollowingcommand:show routing instance node \route ospfOSPFinternalroutesaremarkedwithO infrontoftherouteandexternalroutesaremarkedwithOE.Ifsomeoftheroutesaremissingchecktheconfiguration(seestep3).

3 Check OSPF configuration.TochecktheOSPFconfiguration,followthesesteps:

a) VerifythattheOSPFlicensehasbeeninstalledandthefeatureadminstateisON.Enterthefollowingcommand:show license feature-mgmt name OSPFForRedundancy


Issue:02B DN0976768 49

g OSPFdoesnotstartifthelicenseisnotinstalledorifthelicenseisinstalledbutisinactiveorhasexpired.Alsoifthefeatureadminstateisofforconfig.Ifthefeatureadminstateisoff or config,enterthefollowingcommandtosetittoon:set license feature-mgmt name OSPFForRedundancy feature-admin-state on

IftheOSPFlicenseisnotinstalled,thenenterthefollowingcommandtoinstallthelicense:add license file Where,file specifiesthelocationofOSPFlicense.

b) VerifythatOSPFisenabledonthecorrectinterfaceandOSPFarea.Enterthefollowingcommands:show routing instance node \ospf configIftheOSPFisnotenabledonthecorrectinterfaceandarea,seeConfiguring IPConnection for Multicontroller RNCformoreinformation.

c) VerifythatanIPv4addressissetfortheinterfacewhereOSPFv2isconfigured.Enterthefollowingcommand:show networking addressExampleIftheOSPFv2isconfiguredfortheeth_sfp1interface,enterthefollowingcommand:show networking instance default address owner /IPNIU1-0\iface eth_sfp1Thefollowingoutputisdisplayed:address instance defaultinterfaces : eth_sfp1 type : dedicated address : 200.0.0.1/24 owner : /IPNIU1-0IftheIPv4addressismissing,addittothecorrespondinginterface,seeConfiguring IP Connection for Multicontroller RNCformoreinformation.

d) Verifythattherouter-idisset.Enterthefollowingcommand:show routing instance node \router-idThesystemautomaticallyassignsrouterIDbasedontheIPv4addressesfound.Notethat router-idmus

Troubleshooting Mcrnc

Documents

Transcript of Troubleshooting Mcrnc