DBA in the Trenches

Started testing password changes in our clone environments. Weblogic seems to love throwing you curve balls and when I changes the apps password with the below: FNDCPASS apps/$APPS_PASS 0 Y system/system SYSTEM APPLSYS $APPS_PASS I was not able to start the weblogic managed servers. A look in the log an I found this: weblogic.application.ModuleException:         at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:327)         at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:199)         at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:517)         at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)         at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:159)         at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:45)         at weblogic.applicati

I found this neat little tool for digging into the alert log without having to actually open it up and start searching line by line. Produces a nice little summary and also generates a report if you need to upload something to MOS. adrci> show problem ADR Home = /u03/oratest/testdb/11.2.0.4/admin/TEST_oratest/diag/rdbms/test/TEST: ************************************************************************* PROBLEM_ID           PROBLEM_KEY                                                 LAST_INCIDENT        LASTINC_TIME                              -------------------- ----------------------------------------------------------- -------------------- ----------------------------------------  2                    ORA 600 [pfrdef1_get_tdo#145: pin failed]                   116801               2016-02-10 11:20:29.927000 +02:00        4                    ORA 7445 [kgmdtim()+85]                                     118161               2016-02-10 11:22:02.762000 +02:00    

In R12.2 with Weblogic now playing a huge role as well as java also creating resource heavy processes you will eventually run into a situation where you want to add servers to Weblogic. I'd actually recommend adding at least one more oa_core server as this is the one we've noticed is causing most of the issues. Add oacore_server2 via command line: perl /patch/115/bin/adProvisionEBS.pl ebs-create-managedserver -contextfile=$CONTEXT_FILE -managedsrvname=oacore_server2 -servicetype=oacore -managedsrvport=7205 -logfile=/tmp/addMS_oacoreserver2.log Started the new managed server as follows, admanagedsrvctl.sh start  oacore_server2 Added details of the newly added managed server into the OHS configuration files mod_wl_ohs.conf and apps.conf via command line as below: perl /patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE -configoption=addMS -oacore=testserver.example.com:7205 Bounce the apache server: sh /adapcctl.sh stop sh /adapcctl.sh start Reference: M

Had a few issues come up in the alert log regarding memory with dreaded ORA-600 and ORA-7445. The search tool didn't really show anything regarding the problems so opened an SR and need to attach the incident package. Pretty simple setup and extracting and something I'd like to dig into a bit. Here's how its done: #adrci adrci>show incident -mode detail -p "incident_id=117345" adrci>set homepath diag/rdbms/test/TEST adrci>IPS CREATE PACKAGE INCIDENT 117345 Created package 3 based on incident id 117345, correlation level typical adrci>IPS GENERATE PACKAGE 3 IN /tmp Generated package 3 in file /tmp/ORA7445pe_20160211093814_COM_1.zip, mode complete

This is one of those where I'm left scratching my head. During the clone I was getting issues with the LOCAL_LISTENER parameter. The clone would not complete with this value set. Now when I go and create a new adop session using adop phase=prepare I get the following error [ERROR]     Cannot connect to database [ERROR]     Error Message: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor (DBD ERROR: OCIServerAttach) I found a note to fix the issue and what does it say. I need to set the LOCAL_LISTENER. So I set the listener and now it runs. Again I'm lost for words with this new system of cloning and patching in R12.2 Reference adop phase=prepare fails with: Error Message : ORA-12514: TNS:listener does not currently know of service requested in connect descriptor (Doc ID 2007225.1)

Need to find the location of the alert log file. It's by default in the background dump destination. You can find it in the parameters of the instance. SQL> show parameter background_dump_dest NAME                                 TYPE        VALUE ------------------------------------ ----------- ------------------------------ background_dump_dest                 string      /u01/app/oracle/diag/rdbms/dev/dev/trace SQL>  !ls -ltr /u01/app/oracle/diag/rdbms/dev/dev/trace/al* -rw-r-----. 1 oracle oinstall 134067 Jan 23 10:59 /u01/app/oracle/diag/rdbms/dev/dev/trace/alert_dev.log

Ran into this issue when trying to start up the services in a cloned instance. It took a bit of head scratching and google came to the rescue. Not sure what exactly is happening but I ran this andthen restarted the forms_server1 from weblogic and I was able to login to the instance. admanagedsrvctl.sh start oafm_server1 -Dweblogic.security.allowCryptoJDefaultJCEVerification=true Reference Oracle Forums