SQLRecoverableException: I/O Exception: Connection reset

O

8

40

Yesterday evening I left the office with a running Java program written by me. It should insert a lot of records into our company database (Oracle) using a JDBC connection. This morning when I came back to work I saw this error (caught by a try-catch):

java.sql.SQLRecoverableException: I/O Exception: Connection reset

The program wrote almost all records before getting this problem, but what if it happens early (just minutes after I leave the office at evening)? I cannot understand what happened, I contacted my database admin and he said there was no particular issue on the database.

Any idea on what happened and what can I do do to avoid it?

Odelia answered 24/5, 2011 at 12:29 Comment(0)

C

20

This simply means that something in the backend ( DBMS ) decided to stop working due to unavailability of resources etc. It has nothing to do with your code or the number of inserts. You can read more about similar problems here:

This may not answer your question, but you will get an idea of why it might be happening. You could further discuss with your DBA and see if there is something specific in your case.

Converge answered 24/5, 2011 at 12:41 Comment(3)

I would also recommend a disk space check on your database VM to give your DBA a probable cause to proceed. df -h – Inez 1/7, 2015 at 12:54

none of the links work now. – Palace 21/10, 2022 at 17:33

Also not saved by the Internet Archive. – Jew 4/8, 2023 at 13:44

F

35

The error occurs on some RedHat distributions. The only thing you need to do is to run your application with parameter java.security.egd=file:///dev/urandom:

java -Djava.security.egd=file:///dev/urandom [your command]

Fernery answered 5/9, 2016 at 14:18 Comment(5)

Been racking my brain for 3 days trying to figure this out, and your solution finally fixed it. What an off the wall problem!! Additionally, for a more in depth answer, check out discuss.pivotal.io/hc/en-us/articles/… – Hirsh 12/3, 2018 at 14:48

3 days ? it has been 2 month of headbanging here, much thanks ! (I guess OpenStack entropy throughput is WAY lower than VMware's) – Chewning 25/7, 2018 at 9:55

where do we put this config java -Djava.security.egd=file ? – Endocentric 25/6, 2019 at 12:28

You dont need to put it on any config file. You only need to run your application with the line suggested. – Fernery 1/7, 2019 at 10:48

You may also add the following line in your $JAVA_HOME/jre/lib/security/java.security configuration file: securerandom.source=file:/dev/./urandom Or "systemwide", configure the rngd service with option "-r /dev/urandom" and activate it/run it at boot-time. – Formidable 12/3 at 15:18

S

30

I want to produce a complementary answer of nacho-soriano's solution ...

I recently search to solve a problem where a Java written application (a Talend© ELT job in fact) want to connect to an Oracle database (11g and over) then randomly fail. OS is both RedHat Enterprise and CentOS. Job run very quily in time (no more than half a minute) and occur very often (approximately one run each 5 minutes).

Some times, during night-time as work-time, during database intensive-work usage as lazy work usage, in just a word randomly, connection fail with this message:

Exception in component tOracleConnection_1
java.sql.SQLRecoverableException: Io exception: Connection reset
        at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:101)
        at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:112)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:173)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:229)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:458)
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:411)
        at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:490)
        at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:202)
        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:465)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:208)
    and StackTrace follow ...

##Problem explanation:##

As detailed here

Oracle connection needs some random numbers to assume a good level of security. Linux random number generator produce some numbers bases keyboard and mouse activity (among others) and place them in a stack. You will grant me, on a server, there is not a big amount of such activity. So it can occur that softwares use more random number than generator can produce.

When the pool is empty, reads from /dev/random will block until additional environmental noise is gathered. And Oracle connection fall in timeout (60 seconds by default).

##Solution 1 - Specific for one app solution##

The solution is to give add two parameters given to the JVM while starting:

-Djava.security.egd=file:/dev/./urandom
-Dsecurerandom.source=file:/dev/./urandom

Note: the '/./' is important, do not drop it !

So the launch command line could be:

java -Djava.security.egd=file:/dev/./urandom -Dsecurerandom.source=file:/dev/./urandom -cp <classpath directives> appMainClass <app options and parameters>

One drawback of this solution is that numbers generated are a little less secure as randomness is impacted. If you don't work in a military or secret related industry this solution can be your.

##Solution 2 - General Java JVM solution##

As explained here

Both directives given in solution 1 can be put in Java security setting file.

Take a look at $JAVA_HOME/jre/lib/security/java.security

Change the line

securerandom.source=file:/dev/random

to

securerandom.source=file:/dev/urandom

Change is effective immediately for new running applications.

As for solution #1, one drawback of this solution is that numbers generated are a little less secure as randomness is impacted. This time, it's a global JVM impact. As for solution #1, if you don't work in a military or secret related industry this solution can be your.

We ideally should use "file:/dev/./urandom" after Java 5 as previous path will again point to /dev/random.

Reported Bug : https://bugs.openjdk.java.net/browse/JDK-6202721

##Solution 3 - Hardware solution##

Disclamer: I'm not linked to any of hardware vendor or product ...

If your need is to reach a high quality randomness level, you can replace your Linux random number generator software by a piece of hardware.

Some information are available here.

Regards

Thomas

Souter answered 11/4, 2018 at 12:57 Comment(5)

I just think about it, the solution #4 might be to extend timeout period to go over the 60 seconds limit directly on the database server configuration.<br>In sqlnet.ora and tnsnames.ora Oracle's listener configuration file you can consider TCP.CONNECT_TIMEOUT and SQLNET.OUTBOUND_CONNECT_TIMEOUT parameters. – Souter 11/4, 2018 at 13:0

Can you please explain why you say the '/./' in Solution 1 is important? – Goingover 14/1, 2020 at 15:11

@Goingover He explains it, although I missed it on first read. It's because Java can "replace" /dev/urandom with /dev/random, so the '/./' solution prevents that. – Phallicism 5/1, 2022 at 15:32

Thanks for a great explanation. I was wondering, since the entropy pool is empty, what if we wait for some time, say 5 seconds before starting to write to the database again, would that help? For example we catch the exception SQLRecoverableException in a try/catch block, and execute 'Thread.sleep(5000)` before starting to write to the database again? – Selfpreservation 30/8, 2022 at 20:8

May be a solution, without any guarantee as we hope 5s will be enough Can't try and test your proposal as it's impossible to implement in Talend job :-( – Souter 18/10, 2022 at 13:28

C

20

This simply means that something in the backend ( DBMS ) decided to stop working due to unavailability of resources etc. It has nothing to do with your code or the number of inserts. You can read more about similar problems here:

This may not answer your question, but you will get an idea of why it might be happening. You could further discuss with your DBA and see if there is something specific in your case.

Converge answered 24/5, 2011 at 12:41 Comment(3)

I would also recommend a disk space check on your database VM to give your DBA a probable cause to proceed. df -h – Inez 1/7, 2015 at 12:54

none of the links work now. – Palace 21/10, 2022 at 17:33

Also not saved by the Internet Archive. – Jew 4/8, 2023 at 13:44

U

7

Solution
Change the setup for your application, so you this parameter[-Djava.security.egd=file:/dev/../dev/urandom] next to the java command:

java -Djava.security.egd=file:/dev/../dev/urandom [your command]

Ref :- https://community.oracle.com/thread/943911

Usher answered 30/8, 2016 at 17:38 Comment(0)

Q

5

We experienced these errors intermittently after upgraded from 11g to 12c and our java was on 1.6.

The fix for us was to upgrade java and jdbc from 6 to 7

export JAVA_HOME='/usr/java1.7'

export CLASSPATH=/u01/app/oracle/product/12.1.0/dbhome_1/jdbc/libojdbc7.jar:$CLASSPATH

Several days later, still intermittent connection resets.

We ended up removing all the java 7 above. Java 6 was fine. The problem was fixed by adding this to our user bash_profile.

Our groovy scripts that were experiencing the error were using /dev/random on our batch VM server. Below forced java and groovy to use /dev/urandom.

export JAVA_OPTS=" $JAVA_OPTS -Djava.security.egd=file:///dev/urandom "

Quincunx answered 17/3, 2015 at 18:30 Comment(0)

N

2

Your exception says it all "Connection reset". The connection between your java process and the db server was lost, which could have happened for almost any reason(like network issues). The SQLRecoverableException just means that its recoverable, but the root cause is connection reset.

Nucleoprotein answered 24/5, 2011 at 12:36 Comment(0)

R

1

I had a similar situation when reading from Oracle in a Spark job. This connection reset error was caused by an incompatibility between the Oracle server and the JDBC driver used. Worth checking it.

Redhot answered 30/6, 2020 at 8:52 Comment(0)

W

0

add java security in your run command

java -jar -Djava.security.egd="file:///dev/urandom" yourjarfilename.jar

Wyrick answered 24/9, 2020 at 5:51 Comment(0)

Recommended topics

Hot tags