TROUBLESHOOTING STEPS (Doc ID 730066.1)
Connection Timeout errors ORA-3135 and ORA-3136
A connection timeout error can be issued when an attempt to connect to the database does not complete its connection and authentication phases within the time period allowed by the following:
SQLNET.INBOUND_CONNECT_TIMEOUT and/or INBOUND_CONNECT_TIMEOUT_ server-side parameters.
Starting with Oracle 10.2, the default for these parameters is 60 seconds where in previous releases it was 0, meaning no timeout.
On a timeout, the client program will receive the ORA-3135 (or possibly TNS-3135) error:
ORA-3135 connection lost contact
and the database will log the ORA-3136 error in its alert.log:
... Sat May 10 02:21:38 2008
WARNING: inbound connection timed out (ORA-3136) ...
When a database session is in the authentication phase, it will issue a sequence of SQL statements. The authentication is not complete until all these are parsed, executed, fetched completely. Some of the SQL statements in this list e.g. on 10.2 are:
select value$ from props$ where name = 'GLOBAL_DB_NAME'
select privilege#,level from sysauth$ connect by grantee#=prior privilege#
and privilege#>0 start with grantee#=:1 and privilege#>0
select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'),
SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'SERVICE_NAME'),
INSTANCE_NUMBER, STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN')
from v$instance where INSTANCE_NAME=SYS_CONTEXT('USERENV', 'INSTANCE_NAME')
select privilege# from sysauth$ where (grantee#=:1 or grantee#=1) and privilege#>0
ALTER SESSION SET NLS_LANGUAGE= 'AMERICAN' NLS_TERRITORY= 'AMERICA' NLS_CURRENCY= '$'
NLS_ISO_CURRENCY= 'AMERICA' NLS_NUMERIC_CHARACTERS= '.,' NLS_CALENDAR= 'GREGORIAN'
NLS_DATE_FORMAT= 'DD-MON-RR' NLS_DATE_LANGUAGE= 'AMERICAN' NLS_SORT= 'BINARY' TIME_ZONE= '+02:00'
NLS_COMP= 'BINARY' NLS_DUAL_CURRENCY= '$' NLS_TIME_FORMAT= 'HH.MI.SSXFF AM' NLS_TIMESTAMP_FORMAT=
'DD-MON-RR HH.MI.SSXFF AM' NLS_TIME_TZ_FORMAT= 'HH.MI.SSXFF AM TZR' NLS_TIMESTAMP_TZ_FORMAT=
'DD-MON-RR HH.MI.SSXFF AM TZR'
NOTE: The list of SQL above is not complete and does not represent the ordering of the authentication SQL . Differences may also exist from release to release.
- Hangs during Authentication
The above SQL statements need to be Parsed, Executed and Fetched as happens for all SQL inside an Oracle Database. It follows that any problem encountered during these phases which appears as a hang or severe slow performance may result in a timeout.
Symptoms of such hangs will be seen by the authenticating session as waits for:
• cursor: pin S wait on X
• latch: row cache objects
• row cache lock
Other types of wait events are possible; this list may not be complete.
The issue here is that the authenticating session is blocked waiting to get a shared resource which is held by another session inside the database. That blocker session is itself occupied in a long-running activity (or its own hang) which prevents it from releasing the shared resource needed by the authenticating session in a timely fashion. This results in the timeout being eventually reported to the authenticating session.
- Troubleshooting of Authentication hangs
In such situations, we need to find out the blocker process holding the shared resource needed by the authenticating session in order to see what is happening to it.
Typical diagnostics used in such cases are the following:
- Three consecutive systemstate dumps at level 266 during the time that one or more authenticating sessions are blocked. It is likely that the blocking session will have caused timeouts to more than one connection attempt. Hence, systemstate dumps can be useful even when the time needed to generate them exceeds the period of a single timeout e.g. 60 sec:
$ sqlplus -prelim '/ as sysdba'
oradebug setmypid
oradebug unlimit
oradebug dump systemstate 266
...wait 90 seconds
oradebug dump systemstate 266
...wait 90 seconds
oradebug dump systemstate 266
quit
- ASH reports covering e.g. 10-15 minutes of a time period during which several timeout errors were seen.
- If possible, Two consecutive queries on V$LATCHHOLDER view for the case where the shared resource being waited for is a latch.
select * from v$latchholder;
The systemstate dumps should help in identifying the blocker session.
Level 266 will show us in what code it is executing which may help in locating any existing bug as the root cause.
Examples of issues which can result in Authentication hangs
In some cases, it may be possible to avoid problems with Authentication SQL by pinning such statements in the Shared Pool soon after the instance is started and they are freshly loaded. You can use the following artcile to advise on this:
Document 726780.1 How to Pin a Cursor in the Shared Pool using DBMS_SHARED_POOL.KEEP
Pinning will prevent them from being flushed out due to inactivity and aging and will therefore prevent them for needing to be reloaded in the future i.e. needing to be reparsed and becoming susceptible to Authentication hang issues.
lsnrctl status
to see what it is doing now. – Overton