A Complete Blog for Unix System Administrators

72
A Complete Blog for Unix System Administrators Wednesday, 28 November 2012 Netbackup Important Commands and Explanations Master Server 1) Check the license details /usr/openv/netbackup/bin/admincmd/get_license_key 2) Stop and Start the netabackup services i) /etc/init.d/netbackup stop (start) —> graceful stop and start ii) /usr/openv/netbackup/bin/bp.kill_all —> Stop backup including GUI sessions, ungraceful iii) /usr/openv/netbackup/bin/bp.start_all —> Start the backup iv) /usr/openv/netbackup/bin/initbprd —> starts the master server v) /usr/openv/netbackup/bin/vmd —> starts the media server vi) /usr/openv/netbackup/bin/jnbSA —> Starts the GUI sessions 3) Scan the tape devices #sgscan (in Solaris) #/usr/openv/volmgr/bin/scan (in AIX) 4) Display all the netbackup process #bpps –x 5) Check the backup status In GUI —> Activity monitor In CLI —> #bpdbjobs -report 6) Lists the server errors #bperror

description

A Complete Blog for Unix System Administrators

Transcript of A Complete Blog for Unix System Administrators

A Complete Blog for Unix System AdministratorsWednesday, 28 November 2012Netbackup Important Commands and Explanations Master Server1) Check the license details/usr/openv/netbackup/bin/admincmd/get_license_key

2) Stop and Start the netabackup servicesi) /etc/init.d/netbackup stop (start) > graceful stop and startii) /usr/openv/netbackup/bin/bp.kill_all > Stop backup including GUI sessions, ungracefuliii) /usr/openv/netbackup/bin/bp.start_all > Start the backupiv) /usr/openv/netbackup/bin/initbprd > starts the master serverv) /usr/openv/netbackup/bin/vmd > starts the media servervi) /usr/openv/netbackup/bin/jnbSA > Starts the GUI sessions

3) Scan the tape devices#sgscan (in Solaris)#/usr/openv/volmgr/bin/scan (in AIX)

4) Display all the netbackup process#bpps x

5) Check the backup statusIn GUI > Activity monitorIn CLI > #bpdbjobs -report

6) Lists the server errors#bperror#bperrror U problems hoursago 1#bperror U backstat -by_statcode -hoursago 1

7) Display information about the error code#bperror S

8) Reread bp.conf file without stop/start the netbackup#bprdreq -rereadconfig

Media Server (starts with bpxxx )1) List the storage units#bpstulist U

2) List media details# /usr/openv/netbackup/bin/goodies/available_mediaThis cmd is retrieving all info about media that is available/full/expired/Frozen

3) List all the netbackup jobs#bpdbjobs report

4) Freeze or Unfreeze mediaIn CLI, #bpmedia unfreeze [-freeze] ev

5) List media details#bpmedialist -ev 6) List the media contents#bpmedialist U mcontents m

7) List the information about NB images on media#bpimmedia mediaid -L

8) List backup image information#bpimagelist -U (general)# bpimagelist -media U (for media)

9) Expire a tape# bpexpdate d 0 ev -force

10) Expire client images#bpimage cleanup allclients

11) Which tapes are used for taking backupIn GUI, Backup and Restore > Find the Filesystem > Preview Media ButtonIn CLI, #bpimagelist media U

Volume Commands (starts with vmxxx)

1) Tape Drive (vmoprcmd) 1) List the drive status, detail drive info and pending requestsIn GUI, Device mgmtIn CLI, #vmoprcmd#vmoprcmd d ds (status)#vmopcrmd d pr (pending requests)

2) Control a tape deviceIn GUI, Device mgmtIn CLI, #vmoprcmd [-reset] [-up] [-down]

2) Tape Media commands (vmpool,vmquery,vmchange,vmdelete)1) List all the poolsIn CLI, #vmpool listall -bx

2) List the scratch pool available#vmpool -list_scratch

3) List tapes in poolIn CLI, #vmquery pn -bx4) List all tapes in the robotIn CLI, #vmquery rn 0 bx

5) List cleaning tapesIn CLI, #vmquery mt dlt_clean bx

6) List tape volume details#vmquery m

7) Delete a volume from the catalog#vmdelete m

8) Changes a tapes expiry date#vmchange -exp 12/31/2012 hr:mm:ss m

9) Changes a tapes media pool#vmchange -p -m

3) Tape/Robot commands (starts with tpxxx)1) List the tape drives#tpconfig d

2) List the cleaning times on drives#tpclean -L

3) Clean a drive#tpclean C

Client Commandsi) List the clients#bpplclients

Policy Commandsi) List the policies#bppllist U

ii) List the detailed information about the policies#bpplist U -allpolicies

Netbackup Cheat SheetMaster Server Daemons/ProcessesRequest daemonbprd

Schedulerbpsched (started with bprd)

Netbackup database managerbpdbm (started with bpsched)

Job Monitorbpjobd (started with bpdbm)

Media Server Daemons/ProcessesCommunications daemonbpcd

Backup and restore managerbpbrm (started with bpcd)

Tape Managerbptm (started with bpbrm)

Disk Managerbpdm (started with bpbrm)

Media Managerltid

Bar code readeravrd (started with ltid)

Remote device management/ controls volume databasevmd (started with ltid)

Roboticdaemon (one on each media server) talks to tldcdtldd (started with ltid)

Robotic control daemon talks to the robot directl via scsitldcd (started with ltid)

CatalogsMaster Server

Information about backed-up filesimage - /opt/openv/netbackup/db

Storage Unit, Global Configuration, Catalog backup configuration.config - /opt/openv/netbackup/db

Backup Policy informationclass - /opt/openv/netbackup/db

Job status informationjobs - /opt/openv/netbackup/db

Netbackup logs with error and status informationerror - /opt/openv/netbackup/db

Information on volumes, volume pools, scratch pool and volume groupsvolume - /opt/openv/volmgr/database

Media Server

Tracks assigned volumes (media that has data them)media - /opt/openv/netbackup/db

Information about devices managed by the media serverdevice - /opt/openv/volmgr/database

Log and Information FilesNetbackup and Patch versions/opt/openv/netbackup/bin/version

Media Version/opt/openv/volmgr/version

Patch Level history/opt/openv/netbackup/patch/patch.history

Buffer size/opt/openv/netbackup/db/config/SIZE_DATA_BUFFERS

Number of buffers/opt/openv/netbackup/db/config/NUMBER_DATA_BUFFERS

Network Buffer Size/opt/openv/netbackup/NET_BUFFER_SZ (default = 32)

Java GUI authorisation/opt/openv/java/auth.conf

Catalog type (binary or ASCII)/opt/openv/netbackup/db/config/cat_format.cfg

Netbackup and media manager parameter files/opt/openv/netbackup/bp.conf/opt/openv/volmgr/vm.conf

Corrupt Database image files (5.0 and above)/opt/openv/netbackup/db.corrupt

Server CommandsCheck license details/opt/openv/netbackup/bin/admincmd/get_license_key

Start Netbackupnetbackup start/opt/openv/netbackup/bin/initbprd (master)/opt/openv/volmgr/bin/vmd (media)

Stop Netbackup (does not disconnect GUI sessions)netbackup stop/opt/openv/netbackup/bin/admincmd/bprdreq -terminate (master)/opt/openv/netbackup/bin/bpdbm -terminate (master)

Stop Netbackup and kill all GUI sessions/opt/openv/netbackup/bin/goodies/bp.kill_all

Start the GUI/opt/openv/netbackup/bin/jnbSA

Scan for tape devicessgscan (solaris)ioscan (HPUX)

Display all Netbackup processesbpps -a

lists servers errorsbperror -U -problems -hoursago bperror -U -backstat -by_statcode -hoursago

display information on a error codebperror -statuscode [-recommendation]

Reread bp.conf file without stopping Netbackupbprdreq -rereadconfig

Check database consistencybpdbm -consistency 1bpdbm -consistency 2Check for the below lines:Bad image headerDoes not exist

Netbackup Recovery

Device catalog is intactbprecover -l -m -d dlt (listing)bprecover -r -m -d dlt (recovering)

Device catalog is gone or corruptedbprecover -l -tpath (listing)bprecover -r -tpath (recovering)

Disk backupsbprecover -l -dpath (listing)bprecover -r -dpath (recovering)

Volume CommandsTape Drive and Inventory Commands

List drive status, detail drive info and pending requestsvmoprcmd

List the tape drive statusvmoprcmd -d ds

List the pending requestsvmoprcmd -d pr

Control a tape devicevmoprcmd [-reset][-up][-down]

List all changes in the robot(but do not update)vmupdate -recommend -rt tld -rn 0vmcheckxxx -rt tld -rn 0 -recommend

Empty the robot and re-inventory (using barcodes)vmupdate -rt tld -rn -rh -vh -nostderr -use_barcode_rules -use_seed -empty_ie

Tape Media Commands

List all poolsvmpool -listall -bx

List tapes in poolvmquery -pn -bx

List all tapes in the robotvmquery -rn 0 -bx |grep 'TLD' | sort +4

List cleaning tapesvmquery -mt dlt_clean -bx

List tape volume detailsvmquery -m

Delete a volume from the catalogvmdelete -m

Change a tapes expiry datevmchange -exp 12/31/06 23:59:58 -m

Change a tape's media poolvmchange -p -m

Media commandsList the storage unitsbpstulist -U

Freeze or unfreeze mediabpmedia [-freeze][-unfreeze] -ev

List media detailsbpmedialist -ev

List media contentsbpmedialist -U mcontents -m

List backup Image Informationbpimagelist -backupid

Expire client imagesbpimage -cleanup -allclients

Expire a tapebpexpdate -d 0 -ev -force

List all netbackups jobsbpdbjobs -report [-hoursago]

Move media from one media server to anotherbpmedia -movedb -newserver -oldserver

Tape/Robot commandsList tape drivestpconfig -d

List cleaning times on drivestpclean -L

clean a drivetpclean -C

change a drives cleaning frequencytpclean -F

set a drives cleaning time to zerotpclean -M

Move tapes within robot using robtestrobtestcommands that can be used are as follows:s s (show slots)s d (show drives)s i (show load port)m s250 d5 (move tape from slot 250 into drive 5)uload d5 (unload tape from drive 5)m d5 s250 (move tape from drive 5 to slot 250)m s250 i1 (mov tape from slot 250 to load port 1)

List load port tapesecho "s i q" | tldtest -r /dev/sg/c0t4l0

List all slot contentsecho "s s q" | tldtest -r /dev/sg/c0t4l0

List tape drive contentsecho "s d q" | tldtest -r /dev/sg/c0t4l0

Move a tape in s100 to drive 1echo "m s100 d1" | tldtest -r /dev/sg/c0t4l0

Move a tape to load port 1echo "m s100 i1" | tldtest -r /dev/sg/c0t4l0

Archiving Commandslist archive infobpcatlist -client all -before Jul 01 2006bpcatlist -client all -before Aug 01 2006

archive and remove imagesbpcatlist -before Jul 01 2006 | bpcatarc | bpcatrm

restore archive filesbpcatlist -before Jul 01 2006 | bpcatres

Client commandstest client connectivitybpclntcmd [-ip ]bpclntcmd [-hn ]bpclntcmd [-pn]bpclntcmd [-sv]

veritas netbackup tricks and commandsNetbackup Tips and Tricks (Unix)GlossaryTerm DefinitionCLI Command Line InterfaceGUI Graphical User InterfaceMedia ServerMaster ServerStarting and Stopping NetbackupStopping Netbackup /usr/openv/netbackup/bin/K77netbackup --> graceful shutdown /usr/openv/netbackup/bin/bpps -a --> check for any remaining processes /usr/openv/netbackup/bin/goodies/bp.kill_all ---> kills all remaining netbackup processes, not necessarily graceful /usr/openv/netbackup/bin/bpps -a --> check for any remaining processes kill -9 for any remaining. NOTE: unkillable processes may require a rebootStarting Netbackup /usr/openv/netbackup/bin/S77netbackup --> after bp.kill_all, to restart /usr/openv/volmgr/bin/vmdCommon TasksStarting the Administration GUI java from the windows client x-windows from the server - /usr/openv/netbackup/bin/xnb &Checking Backup Status Activity Monitor or /usr/openv/netbackup/bin/admincmd/bpdbjobs -reportCleaning a tape manually Identify the drive name to be cleanedo tpclean -L Manually clean the drive:o tpclean -CDetermining what tapes were used for a backup GUIo Backup and Restore --> Find the file system --> Preview Media Button CLIo Find the correct backup images bpimagelist -U -client -d -eo Find the media used for those images bpimagelist -U -client -d -e -mediaListing the files in a backupFind the tape(s) used (above procedure using bpimagelist)cd /usr/openv/netbackup/db/jobs/doneRun the following script and redirect it's output to a text file:for file in `grep MOUNTING *|grep |awk '{print $1}'|sed 's/:MOUNTING//'`doecho $filegrep PATH_WRITTEN $file|awk '{print $3}'echo " "echo "==========================================End of Image======================================"echo " "doneThis process works for NBU V3.4: cd /usr/openv/netbackup/db/images/ ls -ltr --> this will identify the directory with the proper date verify directory with "bpdbm -ctime cd ls -ltr --> lists all of the backups for this client on this date cat __.f | awk '{print $10}' --> this prints out the files in the backupFor NBU > V3.4 bpflist --help --> undocumented netbackup command to list files from a binary .f fileInventory the Robot Inventory Robot --> /opt/openv/volmgr/bin/vmcheckxxx -rt robot_type -rn robot_number -list (where robot_type is tld, acs, . . .) Inventory Robot and Update Configuration --> /opt/openv/volmgr/bin/vmupdate -rt robot_type -rn robot_number -list (where robot_type is tld, acs, . . .)Listing Properties of the Volume Pools vmpool -listallScratch Tapes Count scratch tapes: /usr/openv/volmgr/bin/vmquery -pn Scratch | grep -c "robot slot" Moving tapes to the scratch poolo If Needed - Expire the tape bpexpdate -ev -d 0 -force -hosto Move the tape vmchange -p 2 -mChecking Drive Usage /usr/openv/volmgr/bin/vmoprcmdTaking a drive down or up /usr/openv/volmgr/vmoprcmd -down /usr/openv/volmgr/vmoprcmd -upPerforming a Restore From the GUIo user backup & restore --> configuration --> cliento user backup & restore --> configuration --> client to restoreo directory to searcho directory deptho date rangeo file --> browse backups for restoreAdding New Tapes to the Library Using the GUIo Media Management --> Actions --> New --> Single Volume . . -->o Media Type (ie DLT)o Robot Type (ie TLD)o Media ID (from Inventory)o Slot Number (from Inventory)o Robot Number (ie 0)o Volume Groupo Volume Pool (ie Scratch) Using the CLIo vmadd -m -mt -verbose -rt -b -rn -rc1 -p -mm vmpool -listall --> lists all pools, both name and numbero For example: vmadd -m 000151 -mt dlt -verbose -rt tld -b 000151 -rn 0 -rc1 8 -p 2 -mm 0Re-using Tapes from other systems or older Netbackups Expire the mediao bpexpdate -ev MEDIA_ID -d 0 -force -host HOST Deassign the mediao vmquery -deassignbyid MEDIA_ID 4 0 Move to the scratch poolo vmchange -m MEDIA_ID -p POOL# Relabel the mediao bplabel -ev CIM572 -d dlt -p ScratchChanging the attributes of media Changing the barcodeo vmchange -barcode CYM100D -m CYM100 Changing the Volume Poolo vmchange -m MEDIA_ID -p POOL#To expire media bpexpdate -ev -d 0 -force -hostTo unfreeze media List the frozen mediao /usr/openv/netbackup/bin/goodies/available_media | grep -i FROZEN Unfreeze the mediao bpmedia -unfreeze -ev -hTo relabel a tape bplabel -ev -d -p bplabel -ev 000687 -d dlt -p TriVrgt_OFFSITETo remove media from the Netbackup database Verify that there are no images on the tapeo bpimmedia -mediaid 000687 -L Expire the tapeo bpexpdate -ev 000687 -d 0 -host scorpius -force Get the status and pool number of the tapeo vmquery -m 000687 Deassign the tapeo vmquery -deassignbyido vmquery -deassignbyid 000687 4 0x0 Delete the tapeo vmdelete -m 000687Installing the Netbackup Client /update_clients -ForceInstall -ClientList /tmp/clients.lsto requires that TMPDIR and TEMPDIR be set correctlyExcludng files from backup on a client Create /usr/openv/netbackup/exclude_list Put the file specifications of the files/directories to be excludedo /mnt/directory/*Displaying Information about a Tape vmquery -m --> Displays attributes about a particular tape bpmedialist -U -mcontents -ev 000687 --> Displays media contents bpmedialist -U -mlist --> List of all media bpmedialist -U -mlist -ev CYM966 --> Listing of a particular media id bpimmedia -mediaid 000687 -L --> Listing of images on a tapeRobtest Commands Starting robtesto robtesto 1 --> to select TLD 0 Getting helpo ? Looking at contents of the tape driveso s d Looking at the contents of the libraryo s s Moving a tape from a drive to a library sloto s d --> to identify drive number that has tape (Contains Cartridge = yes, Barcode=XXXXXX)o s s --> to identify an empty slot in the tape library (Netbackup will need to be re-inventoried)o m d# s# --> from from drive # to slot #o s d --> verify the tape drive is emptyo s s --> verify the library slot has the tapeConfiguration Files/usr/openv/netbackup/bp.conf configuration file, sets backup server and backup clients force statement must be correct client to browse from client to restore to/usr/openv/volmgr/vmconfLogfilesTo utilize logfiles, create the corresponding directory in /usr/openv/netbackup/logsServer Logfile directories: admin - adminstrative commands bpbrm - backup and restore manager bpcd - client daemon bpdbjobs - database manager program process bpdm - disk manager process bpjava-msvc - Java application server authentication service bpjava-usvc - process that services Java requests bprd - request daemon process bpsched - scheduler process that runs on master servers bptm - tape/optical media management process user-ops - required directory for use by Java programs xbpadm - X based administration utility xbpmon - X based job monitor processClient Logfile directories: bp - client user interface process bparchive - archive program bpbackup - backup program bpbkar - program that generates golden images bpcd - client daemon bpjava-msvc - Java application server authentication service bpjava-usvc - process that services Java requests bplist - program that lists backed up and archived files bpmount - program that determines local mountpoints and wildcard expansion for multiple streams bphdb - Oracle database backup program start process db_log - database specific extension log tar - tar process log during restores user_opsMedia Manager logging automatically goes to the system log using syslogd logging facility.Logging will only occur if these directories are created. These directories will generate a lot of data and should be deleted when no longer necessary.To increase the amount of logging information set VERBOSE=2 in /usr/open/netbackup/bp.conf (default is VERBOSE=1)ProcessesltidacsdvmdUseful Commandsbpcllist - list classesbpclinfo -L --> displays info about a classvmpool - volume poolsvmpool -listallvmpool -listscratchbplabel -ev -d hcartbpbackup db --> backs up the catalogbpclclients --> lists the clients for a particular policy (class)Troubleshootingbperror -statuscode this will bring up drive 0 if it's control shows as down Look for pending requestso /usr/openv/volmgr/bin/vmoprcmd or gui --> device managemento If there is a pending request either re-assign it to a drive, or deny the requestDowned drive does not come back up or does not stay up Check for a hardware problem by looking for messages on the tape library Make sure there is not a tape stuck in the driveo Use robtest (described above) to look at the drives If there is a tape stuck in the drive, try to remove it using robtest If robtest fails, then you must manually remove it.Verify the Client is communicating properly: bpclncmd -ip --> from both client and server bpclntcmd -hn --> from both client and server bpclntcmd -pn --> from client onlyDevice ActionsDevice Management --> info about tape drives dlt hcart (ultrium)Media ActionsMedia id must agree with # of the tapeCreate a media id actions -->new-->single volume-->dlt cart (not dlt2) put it into the "netbackup" volume poolNetbackup ClientTo check things out do this:It could be a couple things. Mostly DNS, bp.conf, or something stupid. Onthe client run this command/usr/openv/netbackup/bin/bpclntcmd -pn/usr/openv/netbackup/bin/bpclntcmd -server "server name"/usr/openv/netbackup/bin/bpclntcmd ip "ip_address"One of these usually fails and your able to fix it right off1074 ./bpclntcmd -hn corpbu11075 ./bpclntcmd -ip 10.194.1.1291076 ping 10.194.1.1291077 ./bpclntcmd -hn corpldv11078 ./bpclntcmd -hn corpbu1.corporate.vox.net1079 ping corpldv11080 ./bpclntcmd -ip 10.194.1.120Must be able to resolve correctly from the master server and the client or it will not work!!!Netbackup CommandMaster Server Daemons/ProcessesRequest daemon bprdScheduler bpsched (started with bprd)Netbackup database manager bpdbm (started with bpsched)Job Monitor bpjobd (started with bpdbm)Media Server Daemons/ProcessesCommunications daemon bpcdBackup and restore manager bpbrm (started with bpcd)Tape Manager bptm (started with bpbrm)Disk Manager bpdm (started with bpbrm)Media Manager ltidBar code reader avrd (started with ltid)Remote device management/ controls volume database vmd (started with ltid)Roboticdaemon (one on each media server) talks to tldcd tldd (started with ltid)Robotic control daemon talks to the robot directl via scsi tldcd (started with ltid)CatalogsMaster ServerInformation about backed-up files image - /opt/openv/netbackup/dbStorage Unit, Global Configuration, Catalog backup configuration. config - /opt/openv/netbackup/dbBackup Policy information class - /opt/openv/netbackup/dbJob status information jobs - /opt/openv/netbackup/dbNetbackup logs with error and status information error - /opt/openv/netbackup/dbInformation on volumes, volume pools, scratch pool and volume groups volume - /opt/openv/volmgr/databaseMedia ServerTracks assigned volumes (media that has data them) media - /opt/openv/netbackup/dbInformation about devices managed by the media server device - /opt/openv/volmgr/databaseLog and Information FilesNetbackup and Patch versions /opt/openv/netbackup/bin/versionMedia Version /opt/openv/volmgr/versionPatch Level history /opt/openv/netbackup/patch/patch.historyBuffer size /opt/openv/netbackup/db/config/SIZE_DATA_BUFFERSNumber of buffers /opt/openv/netbackup/db/config/NUMBER_DATA_BUFFERSNetwork Buffer Size /opt/openv/netbackup/NET_BUFFER_SZ (default = 32)Java GUI authorisation /opt/openv/java/auth.confCatalog type (binary or ASCII) /opt/openv/netbackup/db/config/cat_format.cfgNetbackup and media manager parameter files /opt/openv/netbackup/bp.conf/opt/openv/volmgr/vm.confCorrupt Database image files (5.0 and above) /opt/openv/netbackup/db.corruptServer CommandsCheck license details /opt/openv/netbackup/bin/admincmd/get_license_keyStart Netbackup netbackup start/opt/openv/netbackup/bin/initbprd (master)/opt/openv/volmgr/bin/vmd (media)Stop Netbackup (does not disconnect GUI sessions) netbackup stop/opt/openv/netbackup/bin/admincmd/bprdreq -terminate (master)/opt/openv/netbackup/bin/bpdbm -terminate (master)Stop Netbackup and kill all GUI sessions /opt/openv/netbackup/bin/goodies/bp.kill_allStart the GUI /opt/openv/netbackup/bin/jnbSAScan for tape devices sgscan (solaris)ioscan (HPUX)Display all Netbackup processes bpps -alists servers errors bperror -U -problems -hoursagobperror -U -backstat -by_statcode -hoursagodisplay information on a error code bperror -statuscode [-recommendation]Reread bp.conf file without stopping Netbackup bprdreq -rereadconfigCheck database consistency bpdbm -consistency 1bpdbm -consistency 2Check for the below lines:Bad image headerDoes not existNetbackup RecoveryDevice catalog is intact bprecover -l -m -d dlt (listing)bprecover -r -m -d dlt (recovering)Device catalog is gone or corrupted bprecover -l -tpath (listing)bprecover -r -tpath (recovering)Disk backups bprecover -l -dpath (listing)bprecover -r -dpath (recovering)Volume CommandsTape Drive and Inventory CommandsList drive status, detail drive info and pending requests vmoprcmdList the tape drive status vmoprcmd -d dsList the pending requests vmoprcmd -d prControl a tape device vmoprcmd [-reset][-up][-down]List all changes in the robot(but do not update) vmupdate -recommend -rt tld -rn 0vmcheckxxx -rt tld -rn 0 -recommendEmpty the robot and re-inventory (using barcodes) vmupdate -rt tld -rn -rh -vh -nostderr -use_barcode_rules -use_seed -empty_ieTape Media CommandsList all pools vmpool -listall -bxList tapes in pool vmquery -pn -bxList all tapes in the robot vmquery -rn 0 -bx |grep 'TLD' | sort +4List cleaning tapes vmquery -mt dlt_clean -bxList tape volume details vmquery -mDelete a volume from the catalog vmdelete -mChange a tapes expiry date vmchange -exp 12/31/06 23:59:58 -mChange a tape's media pool vmchange -p -mMedia commandsList the storage units bpstulist -UFreeze or unfreeze media bpmedia [-freeze][-unfreeze] -evList media details bpmedialist -evList media contents bpmedialist -U mcontents -mList backup Image Information bpimagelist -backupid Expire client images bpimage -cleanup -allclientsExpire a tape bpexpdate -d 0 -ev -forceList all netbackups jobs bpdbjobs -report [-hoursago]Move media from one media server to another bpmedia -movedb -newserver -oldserverTape/Robot commandsList tape drives tpconfig -dList cleaning times on drives tpclean -Lclean a drive tpclean -Cchange a drives cleaning frequency tpclean -Fset a drives cleaning time to zero tpclean -MMove tapes within robot using robtest robtestcommands that can be used are as follows:s s (show slots)s d (show drives)s i (show load port)m s250 d5 (move tape from slot 250 into drive 5)uload d5 (unload tape from drive 5)m d5 s250 (move tape from drive 5 to slot 250)m s250 i1 (mov tape from slot 250 to load port 1)List load port tapes echo "s i q" | tldtest -r /dev/sg/c0t4l0List all slot contents echo "s s q" | tldtest -r /dev/sg/c0t4l0List tape drive contents echo "s d q" | tldtest -r /dev/sg/c0t4l0Move a tape in s100 to drive 1 echo "m s100 d1" | tldtest -r /dev/sg/c0t4l0Move a tape to load port 1 echo "m s100 i1" | tldtest -r /dev/sg/c0t4l0Archiving Commandslist archive info bpcatlist -client all -before Jul 01 2006bpcatlist -client all -before Aug 01 2006archive and remove images bpcatlist -before Jul 01 2006 | bpcatarc | bpcatrmrestore archive files bpcatlist -before Jul 01 2006 | bpcatresClient commandstest client connectivity bpclntcmd [-ip ]bpclntcmd [-hn ]bpclntcmd [-pn]bpclntcmd [-sv]1.To expire the taperakshak:/usr/openv/netbackup/bin/admincmd [23] > ./bpexpdate -m SVOB06 -d 0rakshak:/usr/openv/volmgr/bin [40] > ./vmdelete -m SVOB02/var/adm [1] >> media/var/adm [2] >> bpps -aNB Processes------------root 536 1 0 Dec 23 ? 0:00 /bin/sh /usr/openv/netbackup/bin/admincmd/nbdbdmon --user=rootroot 548 536 0 Dec 23 ? 0:00 /usr/openv/db/bin/nbdbd --basedir=/usr/openv/db --datadir=/usr/openv/db/var --uroot 441 1 0 Dec 23 ? 0:17 /usr/openv/netbackup/bin/bprdroot 476 1 0 Dec 23 ? 3:23 /usr/openv/netbackup/bin/bpdbmroot 12691 441 0 23:30:19 ? 0:12 /usr/openv/netbackup/bin/bpsched -ppid 441/var/adm [12] >> bpps -a to kill the GUI of rakshakroot 11142 11120 0 09:25:25 pts/4 1:06 /usr/openv/java/jre/bin/java -Dvrts.NBJAVA_CONF=/usr/openv/java/nbj.conf -Djavaroot 27301 27279 0 Jan 11 pts/2 5:31 /usr/openv/java/jre/bin/java -Dvrts.NBJAVA_CONF=/usr/openv/java/nbj.conf -DjavaTo check the media list/usr/openv/netbackup/bin/admincmd [15] > mediaNetbackup rebootHi Jerome,Following timing where I am going to run the backup: ( Please check the below ones, because it's full backup and I have not run from last 2 months )ZIN05VIEW01:10:15 Sat morningZIN05VIEW02:10:30 Sat morningZIN05VIEW03:10:45 Sat morningRakshak:23:00 Sat nightZIN05VOB03:23:00 Sat NightCheck the below steps , If you rebooting the vobserver and veiew servers.A. Telnet to rakshakB. Set the displayC. Go to /usr/openv/netbackup/bin and then run ./jnbSA &D. Gui will open then on the password tab give the rakshak root passowrd.E. Check any active job is running by clicking activity monitor.F. If at all any jobs pertaining to critical servers are running, then select the jobs and cancel.If you are rebootingVIEW01VIEW02VIEW03VOB03Then check the follwing process are running or not1. Go to /usr/openv/volmgr/bin/sgscanzin05view01:/usr/openv/volmgr/bin # ./sgscan##WARNING: detected StorEdge Network Foundation connected devices not in# SG configuration file:## Device World Wide Port Name 21000004cf277475# Device World Wide Port Name 21000004cf9b645a# Device World Wide Port Name 21000004cfa17ac8# Device World Wide Port Name 5080020000197169# Device World Wide Port Name 21000004cfa17b08# Device World Wide Port Name 500000e010136591# Device World Wide Port Name 21000004cfa17211# Device World Wide Port Name 21000004cf9b61fb# Device World Wide Port Name 21000004cf9b6ada# Device World Wide Port Name 21000004cf9b6aee# Device World Wide Port Name 21000004cf9b6898# Device World Wide Port Name 21000004cf9b61df# Device World Wide Port Name 22000004cf9b645a# Device World Wide Port Name 22000004cf277475# Device World Wide Port Name 22000004cfa17ac8# Device World Wide Port Name 508002000019716a# Device World Wide Port Name 22000004cf9b61fb# Device World Wide Port Name 22000004cfa17211# Device World Wide Port Name 22000004cf9b6ada# Device World Wide Port Name 22000004cf9b6aee# Device World Wide Port Name 22000004cf9b6898# Device World Wide Port Name 22000004cf9b61df# Device World Wide Port Name 22000004cfa17b08# Device World Wide Port Name 500000e010136592## See /usr/openv/volmgr/MediaMgr_DeviceConfig_Guide.txt chapter# "Special configuration for "Sun StorEdge Network Foundation" HBA/Driver"# for information on how to use sg.build and sg.install to# configure these devices#/dev/sg/c0tw2100001086103bacl0: Changer: "ATL M2500"/dev/sg/c0tw2100001086103bacl1: Tape (/dev/rmt/3): "QUANTUM SDLT320"/dev/sg/c0tw2100001086103bacl2: Tape (/dev/rmt/2): "QUANTUM SDLT320"/dev/sg/c0tw2100001086104420l0: Tape (/dev/rmt/1): "QUANTUM SDLT320"/dev/sg/c0tw2100001086104420l1: Tape (/dev/rmt/0): "QUANTUM SDLT320"If it's not showing then do/etc/rc2.d/S99M2500 startAgain run the sgscan command2. go to /usropenv/netbackup/bin/bpps -aNB Processes------------root 27315 1 0 Sep 22 ? 0:00 /bin/sh /usr/openv/netbackup/bin/admincmd/nbdbdmon --user=rootroot 27327 27315 0 Sep 22 ? 0:00 /usr/openv/db/bin/nbdbd --basedir=/usr/openv/db --datadir=/usr/openv/db/var --uMM Processes------------root 29066 28896 0 Sep 22 ? 0:00 avrdroot 27238 1 0 Sep 22 ? 0:01 vmdroot 28896 1 0 Sep 22 ? 0:02 /usr/openv/volmgr/bin/ltidroot 29064 28896 0 Sep 22 ? 0:00 tlddroot 526 27238 0 Sep 22 ? 0:01 rdevmi -sockfd 6 -rroot 29903 27238 0 Sep 22 ? 0:01 rdevmi -sockfd 6 -rroot 66 27238 0 Sep 22 ? 0:01 rdevmi -sockfd 6 -rroot 15380 27238 0 11:07:44 ? 0:00 rdevmi -sockfd 6 -rIf it's not running then go to the below path to start the netbackup service./usr/openv/netbackup/bin/goodies/netbackup3. On vob03 check vxtask list whether the flashsnap is running or not.Hope you will understand my problem....Regards,Ragha.....To freeze the tapes you need to login to the respective media serverscd /usr/openv/netbackup/bin/admincmdzin05view03:/usr/openv/netbackup/bin/admincmd # ./bpmedia -freeze -m SVEW03To check the media list 2 methods1. avilable media it will displays only what all the media available and which media which belongs2.bpmedialisd give all the information about what all the media availbles and media which belongs to the pools and when it allocated and expiration last date updated and read etc.. with respect to media servers.1.rakshak:/ [1] > cd /usr/openv/netbackup/bin/admincmd/rakshak:/usr/openv/netbackup/bin/admincmd [2] > ./bpmedialistServer Host = rakshakid rl images allocated last updated density kbytes restoresvimages expiration last read --------------------------------------------------------------------------------DMON01 8 100 06/14/2004 15:13 N/A dlt2 2212471 0MPX 10/19/2005 16:19 11/03/2004 19:13 IMPORTEDSDEL01 24 395 04/23/2004 09:45 12/09/2004 17:29 dlt2 211914147 2MPX 12/07/2014 17:29 12/29/2004 12:18SDLY01 5 2611 08/26/2004 21:05 01/12/2005 21:57 dlt2 323247463 0Server Host = zin05view01id rl images allocated last updated density kbytes restoresvimages expiration last read --------------------------------------------------------------------------------SVEW02 8 3 12/31/2004 23:37 01/12/2005 23:30 dlt2 192245499 0MPX 01/12/2006 23:30 N/ASVEW06 8 317 12/22/2004 11:29 12/31/2004 23:37 dlt2 421554297 0MPX 12/31/2005 23:37 N/A FULLServer Host = zin05view02id rl images allocated last updated density kbytes restoresvimages expiration last read --------------------------------------------------------------------------------SVEW01 8 3 01/01/2005 02:40 01/12/2005 23:54 dlt2 225510259 0MPX 01/12/2006 23:54 N/ASVEW13 8 229 12/18/2004 01:02 01/01/2005 02:40 dlt2 398799500 0MPX 01/01/2006 02:40 N/A FULLSVEW14 8 210 12/18/2004 01:02 12/25/2004 01:37 dlt2 408836258 0MPX 12/25/2005 01:37 N/A FULLSVEW15 8 1 12/25/2004 01:37 12/25/2004 01:37 dlt2 13287112 0Server Host = zin05view03id rl images allocated last updated density kbytes restoresvimages expiration last read --------------------------------------------------------------------------------SVEW03 8 2 01/11/2005 09:23 01/13/2005 00:55 dlt2 101592944MPX 01/13/2006 00:55 N/ASVEW10 8 14 12/18/2004 11:53 12/18/2004 12:04 dlt2 18070585 0MPX 12/18/2005 12:11 N/ASVEW11 8 44 12/18/2004 11:02 01/11/2005 09:23 dlt2 391753568 0MPX 01/11/2006 09:23 N/A FULLPrerequisitesBefore attending this course, you should have: * One year of NetBackup UNIX administration * Two years experience as a UNIX system administrator (including networking) * Familiarity with backup and restore concepts and SAN conceptsSyllabusCourse ContentUpgrading NetBackup Preparing for an Upgrade Upgrading to NetBackup 6.xAnalyzing and Detecting Problems Analyzing Problems Detecting Problems Using the Activity Monitor and Reports Detecting Problems Using NBSupport Detecting Problems Using E-mail Notification Detecting Problems Using NOMProcess Flow NetBackup Architecture and Processes Review Backup Process Flow Restore Process FlowUsing Debug Logs Log Management Capturing Debug Logs Viewing LogsNetBackup Database Troubleshooting The NetBackup Relational Database (NBDB) NBDB Creation Process The EMM Domain EMM Server Backup Process Flow Protecting the NBDB Maintaining the NBDBTroubleshooting Devices Device Error Status Codes DOWN Drives: Isolating Tape Drive Problems Troubleshooting Shared Storage Tape Alert Drive DiagnosticsTroubleshooting Media Media-Related Status Codes Troubleshooting Frozen and Suspended MediaTroubleshooting Network Issues PBX and VNETD Usage NetBackup and TCP/IP Communications Communication Errors Physical Network Failures NetBackup Configuration Errors (?)Performance Tuning Performance Concepts Isolating Bottlenecks Addressing Bottlenecks Netbackup Cheat Sheet Master Daemons | Media Daemons | Catalog | Log Files | Server | Volume | Media | Tape/Robot | Archive | Client Master Server Daemons/ProcessesRequest daemon bprd

Schedulerbpsched (started with bprd)

Netbackup database manager bpdbm (started with bpsched)

Job Monitor bpjobd (started with bpdbm)

Media Server Daemons/ProcessesCommunications daemon bpcd

Backup and restore managerbpbrm (started with bpcd)

Tape Manager bptm (started with bpbrm)

Disk Manager bpdm (started with bpbrm)

Media Manager ltid

Bar code reader avrd (started with ltid)

Remote device management/ controls volume database vmd (started with ltid)

Roboticdaemon (one on each media server) talks to tldcd tldd (started with ltid)

Robotic control daemon talks to the robot directl via scsi tldcd (started with ltid)

CatalogsMaster Server

Information about backed-up files image - /opt/openv/netbackup/db

Storage Unit, Global Configuration, Catalog backup configuration. config - /opt/openv/netbackup/db

Backup Policy information class - /opt/openv/netbackup/db

Job status information jobs - /opt/openv/netbackup/db

Netbackup logs with error and status information error - /opt/openv/netbackup/db

Information on volumes, volume pools, scratch pool and volume groups volume - /opt/openv/volmgr/database

Media Server

Tracks assigned volumes (media that has data them) media - /opt/openv/netbackup/db

Information about devices managed by the media serverdevice - /opt/openv/volmgr/database

Log and Information FilesNetbackup and Patch versions /opt/openv/netbackup/bin/version

Media Version /opt/openv/volmgr/version

Patch Level history/opt/openv/netbackup/patch/patch.history

Buffer size /opt/openv/netbackup/db/config/SIZE_DATA_BUFFERS

Number of buffers /opt/openv/netbackup/db/config/NUMBER_DATA_BUFFERS

Network Buffer Size/opt/openv/netbackup/NET_BUFFER_SZ (default = 32)

Java GUI authorisation /opt/openv/java/auth.conf

Catalog type (binary or ASCII)/opt/openv/netbackup/db/config/cat_format.cfg

Netbackup and media manager parameter files /opt/openv/netbackup/bp.conf/opt/openv/volmgr/vm.conf

Corrupt Database image files (5.0 and above) /opt/openv/netbackup/db.corrupt

Server CommandsCheck license details /opt/openv/netbackup/bin/admincmd/get_license_key

Start Netbackup netbackup start

/opt/openv/netbackup/bin/initbprd (master)/opt/openv/volmgr/bin/vmd (media)

Stop Netbackup (does not disconnect GUI sessions) netbackup stop /opt/openv/netbackup/bin/admincmd/bprdreq -terminate (master)/opt/openv/netbackup/bin/bpdbm -terminate (master)

Stop Netbackup and kill all GUI sessions /opt/openv/netbackup/bin/goodies/bp.kill_all

Start the GUI /opt/openv/netbackup/bin/jnbSA

Scan for tape devicessgscan (solaris)ioscan (HPUX)

Display all Netbackup processes bpps -a

lists servers errors bperror -U -problems -hoursago bperror -U -backstat -by_statcode -hoursago

display information on a error code bperror -statuscode [-recommendation]

Reread bp.conf file without stopping Netbackupbprdreq -rereadconfig

Check database consistency bpdbm -consistency 1bpdbm -consistency 2Check for the below lines:Bad image headerDoes not exist

Netbackup Recovery

Device catalog is intactbprecover -l -m -d dlt (listing)bprecover -r -m -d dlt (recovering)

Device catalog is gone or corrupted bprecover -l -tpath (listing)bprecover -r -tpath (recovering)

Disk backups bprecover -l -dpath (listing)bprecover -r -dpath (recovering)

Volume CommandsTape Drive and Inventory Commands

List drive status, detail drive info and pending requests vmoprcmd

List the tape drive status vmoprcmd -d ds

List the pending requests vmoprcmd -d pr

Control a tape device vmoprcmd [-reset][-up][-down]

List all changes in the robot(but do not update)vmupdate -recommend -rt tld -rn 0vmcheckxxx -rt tld -rn 0 -recommend

Empty the robot and re-inventory (using barcodes) vmupdate -rt tld -rn -rh -vh -nostderr -use_barcode_rules -use_seed -empty_ie

Tape Media Commands

List all pools vmpool -listall -bx

List tapes in pool vmquery -pn -bx

List all tapes in the robot vmquery -rn 0 -bx |grep 'TLD' | sort +4

List cleaning tapes vmquery -mt dlt_clean -bx

List tape volume details vmquery -m

Delete a volume from the catalog vmdelete -m

Change a tapes expiry date vmchange -exp 12/31/06 23:59:58 -m

Change a tape's media pool vmchange -p -m

Media commandsList the storage units bpstulist -U

Freeze or unfreeze media bpmedia [-freeze][-unfreeze] -ev

List media details bpmedialist -ev

List media contents bpmedialist -U mcontents -m

List backup Image Information bpimagelist -backupid

Expire client images bpimage -cleanup -allclients

Expire a tape bpexpdate -d 0 -ev -force

List all netbackups jobs bpdbjobs -report [-hoursago]

Move media from one media server to another bpmedia -movedb -newserver -oldserver

Tape/Robot commandsList tape drivestpconfig -d

List cleaning times on drives tpclean -L

clean a drive tpclean -C

change a drives cleaning frequency tpclean -F

set a drives cleaning time to zero tpclean -M

Move tapes within robot using robtest robtestcommands that can be used are as follows:

s s (show slots)s d (show drives)s i (show load port)m s250 d5 (move tape from slot 250 into drive 5)uload d5 (unload tape from drive 5) m d5 s250 (move tape from drive 5 to slot 250)m s250 i1 (mov tape from slot 250 to load port 1)

List load port tapes echo "s i q" | tldtest -r /dev/sg/c0t4l0

List all slot contents echo "s s q" | tldtest -r /dev/sg/c0t4l0

List tape drive contents echo "s d q" | tldtest -r /dev/sg/c0t4l0

Move a tape in s100 to drive 1 echo "m s100 d1" | tldtest -r /dev/sg/c0t4l0

Move a tape to load port 1 echo "m s100 i1" | tldtest -r /dev/sg/c0t4l0

Archiving Commands list archive info bpcatlist -client all -before Jul 01 2006bpcatlist -client all -before Aug 01 2006

archive and remove images bpcatlist -before Jul 01 2006 | bpcatarc | bpcatrm

restore archive files bpcatlist -before Jul 01 2006 | bpcatres

Client commands test client connectivitybpclntcmd [-ip ]bpclntcmd [-hn ]bpclntcmd [-pn]bpclntcmd [-sv]

Daily tasks of Backup AdminsIn this article I would like to share what Symantec consulting recommends that Netbackup Administrator perform the following tasks to ensure a properly funtioning Netbackup domain.This information has been taken from from Symantec Netbackup Configuration Report document.1. Monitor all NetBackup daemons are running on the master and all media servers. This can be monitored from the Console > Activity Monitor >Daemons.2. Monitor all NetBackup processes are running on the master and all media servers. This can be done from the Console > Activity Monitor > Processes tab.3. Monitor all backup jobs and investigate any failed jobs. This can be done from the Console > Activity Monitor > Jobs tab.4. Monitor the health of all backup devices. This can be done from the Console > Media and Device Management > Device Monitor5. If the drives are in a down or unknown state, investigate the causes through system logs or Event Viewer logs. Downed drives can be upped from Console > Media and Device Management > Device Monitor > select one or multiple drives > Actions > select Up Drive.6. If the drives are in an unknown state, reset the drive to NetBackups known state from Console > Media and Device Management > Device Monitor > select one or more device > Actions > Reset Drive.7. Resolve any pending tape mount requests for restores or backups. When NetBackup cannot automatically service a tape mount request due to a particular media not being available, operator action is required to complete the request. This can be seen from Console > Device and Media Management > Device Monitor > Drives Tab. After the requested media is inserted into an available drive, from the Actions > select Assign Request.8. Monitor availability of Media in the scratch pool for the daily backup operations. This can be done from the Console > Media and Device Management > Media and selecting the Scratch Pool.9. Pre-labeling media will speed up mounting of tapes during backups. To pre-label tapes, from the Console > Media and Device Management > Media > select the volumes to be labeled on the right pane > Actions > select Label and check Verify media label before operation.10. Monitor for Frozen and inactive media. This can be done from Console > NetBackup Management > Reports > Tape Reports > Tape Summary. A frozen media is unavailable for backups and never expires. To unfreeze the media, from the Console > Media and Device Management > Media > select the media to be unfreezed and Actions > Unfreeze. 11. Monitor the NetBackup Problems Reports or All Log Enteries reports for NetBackup Errors. This can be done from Console > NetBackup Management > Reports > Problems or All Log Entries. The Troubleshooter can be launched by selecting a particular line in the reports pane and right clicking to get additional menu items as shown below. Troubleshooter provides explanation and tips for resolutions of the problems.1. Monitor the disk space for NetBackup Catalogs and log files. If necessary add additional space before the file system gets full.2. Periodically run the check_coverage script from the /usr/openv/netbackup/bin/goodies directory to verify that there are no gaps in client file system backups.Performance metrics of Symantec Netbackup ServerSymantec consulting recommends the following metrics be tracked on a regular basis to ensure an optimally performing NetBackup infrastructure.Master Server:1. CPU load: Recommend that it be less than 80% throughout the backup window.2. Memory usage: Less than 90% usage all through the backup window.3. Catalog Disk Space: Less than 90% full.4. Daily Backup Volumes: Trends point to impending problems with the capacity of the infrastructure.5. Backup Jobs outside the window: Points to schedule adjustments.6. Backup Failures: Recommend less than 5%.7. Backup Failure Code Trends: Points to other environmental problems in the enterprise.8. Media in the scratch pool: Recommended to have at least twice the number of media required for a weekly full.9. Restore Start Times: Points to tape drive resource contentions.Media Servers:1. CPU load: Recommended to be less than 90%2. Memory usage: Recommended to be less than 90%3. HBA Saturation: Recommended to be greater than 85%4. NIC Saturation: Recommended to be greater than 70%5. Tape Drive Utilization: Recommended to be evenly used across all tape drivesPlease refer the following link to access the document.https://symiqforpartners.com/sites/sdrk/dp/Service%20Offering%20Library/Supplement_NBU_ConfigReport_Implementation_A4_v1.00.docx