The customer alerted us that the scripted RMAN backups for smstrain were failing. I confirmed this by looking in the log …
oracle> cd /mserv/oracle/logs/RMAN
oracle> ls -lrt *smstrain*
oracle> pg sms_rmanhot_smstrain_Thu.log
…
Starting backup at 04-JUN-09 05:10:28
current log archived
released channel: t1
RMAN-00571:
===========================================================
RMAN-00569: =============== ERROR MESSAGE
STACK FOLLOWS ===============
RMAN-00571:
===========================================================
RMAN-03002: failure of backup plus archivelog
command at 06/04/2009 05:14:05
RMAN-06059: expected archived log not found, lost of archived log compromises recoverability
ORA-19625: error identifying file /dbsms/smstrain/oradata/smstrain/archive/arch_641734909_1_6050.arc
ORA-27037: unable to obtain file status
SVR4 Error: 2: No such file or directory
Additional information:
3
…
… which seems to say that an expected archive-log was not found. I guess another backup deleted it, or perhaps it was ‘tidied away’ to save space.
The fix was to manually resync rmans catalog for the instance …
oracle> .oraenv
oracle> smstrain
oracle> rman target /
rman> crosscheck archivelog all;
rman> exit
This prompted me to a) resync all instances on the server, and b) run a full backup on each.