broken pipe exception with MySql in Grails app
Asked Answered
N

4

12

I have deployed a Grails 2.2.1 application to a remote server. After a few hours, when I access to home page of the application, I see the following exception:

Error 500: Internal Server Error

URI
    /DocGemStudioZoccali/
Class
    java.net.SocketException
Message
    Broken pipe

Trace

    Line | Method
->> 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
|    615 | run       in java.util.concurrent.ThreadPoolExecutor$Worker
^    722 | run . . . in java.lang.Thread

Caused by GroovyPagesException: Error processing GroovyPageView: could not execute query; nested exception is org.hibernate.exception.JDBCConnectionException: could not execute query
->>  631 | runWorker in /index.gsp
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

Caused by DataAccessResourceFailureException: could not execute query; nested exception is org.hibernate.exception.JDBCConnectionException: could not execute query
->>  108 | doCall    in org.grails.datastore.gorm.GormStaticApi$_methodMissing_closure2
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
|     25 | doCall    in gsp_docGemStudioZoccaliindex_gsp$_run_closure1
|     38 | run . . . in gsp_docGemStudioZoccaliindex_gsp
|   1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
|    615 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^    722 | run       in java.lang.Thread

Caused by JDBCConnectionException: could not execute query
->>  108 | doCall    in org.grails.datastore.gorm.GormStaticApi$_methodMissing_closure2
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
|     25 | doCall    in gsp_docGemStudioZoccaliindex_gsp$_run_closure1
|     38 | run . . . in gsp_docGemStudioZoccaliindex_gsp
|   1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
|    615 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^    722 | run       in java.lang.Thread

Caused by CommunicationsException: The last packet successfully received from the server was 138,684,859 milliseconds ago.  The last packet sent successfully to the server was 138,684,859 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
->>  411 | handleNewInstance in com.mysql.jdbc.Util
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
|   1116 | createCommunicationsException in com.mysql.jdbc.SQLError
|   3851 | send . .  in com.mysql.jdbc.MysqlIO
|   2471 | sendCommand in     ''
|   2651 | sqlQueryDirect in     ''
|   2683 | execSQL   in com.mysql.jdbc.ConnectionImpl
|   2144 | executeInternal in com.mysql.jdbc.PreparedStatement
|   2310 | executeQuery in     ''
|     96 | executeQuery in org.apache.commons.dbcp.DelegatingPreparedStatement
|    108 | doCall    in org.grails.datastore.gorm.GormStaticApi$_methodMissing_closure2
|     25 | doCall .  in gsp_docGemStudioZoccaliindex_gsp$_run_closure1
|     38 | run       in gsp_docGemStudioZoccaliindex_gsp
|   1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
|    615 | run       in java.util.concurrent.ThreadPoolExecutor$Worker
^    722 | run . . . in java.lang.Thread

Caused by SocketException: Broken pipe
->>  109 | socketWrite in java.net.SocketOutputStream
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
|    153 | write     in     ''
|     82 | flushBuffer in java.io.BufferedOutputStream
|    140 | flush     in     ''
|   3832 | send . .  in com.mysql.jdbc.MysqlIO
|   2471 | sendCommand in     ''
|   2651 | sqlQueryDirect in     ''
|   2683 | execSQL   in com.mysql.jdbc.ConnectionImpl
|   2144 | executeInternal in com.mysql.jdbc.PreparedStatement
|   2310 | executeQuery in     ''
|     96 | executeQuery in org.apache.commons.dbcp.DelegatingPreparedStatement
|    108 | doCall    in org.grails.datastore.gorm.GormStaticApi$_methodMissing_closure2
|     25 | doCall .  in gsp_docGemStudioZoccaliindex_gsp$_run_closure1
|     38 | run       in gsp_docGemStudioZoccaliindex_gsp
|   1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
|    615 | run       in java.util.concurrent.ThreadPoolExecutor$Worker
^    722 | run . . . in java.lang.Thread

I've tried to solve the problem as seen in another question in SO, but it does not work for me.

Here is what I've done inside config file:

dataSource {
        dbCreate = "update"
        url = "jdbc:mysql://urlofapp/myappname?useUnicode=yes&characterEncoding=UTF-8"
        username = "root"
        password = "root"
        //run the evictor every 30 minutes and evict any connections older than 30 minutes.
        minEvictableIdleTimeMillis=1800000
        timeBetweenEvictionRunsMillis=1800000
        numTestsPerEvictionRun=3
        //test the connection while its idle, before borrow and return it
        testOnBorrow=true
        testWhileIdle=true
        testOnReturn=true
        validationQuery="SELECT 1"
    }

Notice that, if I refresh the page, everything works correctly. Anybody knows something to solve this issue?

EDIT 1:

I've edited the my.cnf file in /etc folder, adding the following:

wait_timeout = 28800
interactive_timeout = 28800

but nothing changes

EDIT 2:

This is the actual data source that I'm using. Broken pipe still appears

dataSource {
   dbCreate = "update"
   url = "jdbc:mysql://urlofapp/myappname?useUnicode=yes&characterEncoding=UTF-8&autoReconnect=true"
   username = "root"
   password = "root"
   minIdle = 5
   maxIdle = 25
   maxWait = 10000
   maxAge = 10 * 60000
   timeBetweenEvictionRunsMillis = 5000
   minEvictableIdleTimeMillis = 60000
   validationQuery = "SELECT 1"
   validationQueryTimeout = 3
   validationInterval = 15000
   testOnBorrow = true
   testWhileIdle = true
   testOnReturn = false
   jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
   defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}

EDIT 3:

Here is the stacktrace after the suggestion given by James Kleeh

2015-12-04 17:41:36,924 [http-bio-8080-exec-63] ERROR context.GrailsContextLoader  - Error initializing the application: Error creating bean with name 'transactionManagerPostProcessor': Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionManager': Cannot resolve reference to bean 'sessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory': Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Cannot resolve reference to bean 'dataSourceUnproxied' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceUnproxied': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionManagerPostProcessor': Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionManager': Cannot resolve reference to bean 'sessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory': Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Cannot resolve reference to bean 'dataSourceUnproxied' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceUnproxied': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionManager': Cannot resolve reference to bean 'sessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory': Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Cannot resolve reference to bean 'dataSourceUnproxied' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceUnproxied': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
        ... 3 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory': Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Cannot resolve reference to bean 'dataSourceUnproxied' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceUnproxied': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
        ... 3 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Cannot resolve reference to bean 'dataSourceUnproxied' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceUnproxied': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
        ... 3 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceUnproxied': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
        ... 3 more
Caused by: org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
        ... 3 more
2015-12-04 17:41:36,931 [http-bio-8080-exec-63] ERROR context.GrailsContextLoader  - Error initializing Grails: Error creating bean with name 'transactionManagerPostProcessor': Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionManager': Cannot resolve reference to bean 'sessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory': Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Cannot resolve reference to bean 'dataSourceUnproxied' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceUnproxied': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionManagerPostProcessor': Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionManager': Cannot resolve reference to bean 'sessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory': Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Cannot resolve reference to bean 'dataSourceUnproxied' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceUnproxied': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionManager': Cannot resolve reference to bean 'sessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory': Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Cannot resolve reference to bean 'dataSourceUnproxied' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceUnproxied': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
        ... 3 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory': Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Cannot resolve reference to bean 'dataSourceUnproxied' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceUnproxied': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
        ... 3 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Cannot resolve reference to bean 'dataSourceUnproxied' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceUnproxied': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
        ... 3 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSourceUnproxied': Error setting property values; nested exception is org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
        ... 3 more
Caused by: org.springframework.beans.NotWritablePropertyException: Invalid property 'maxAge' of bean class [org.apache.commons.dbcp.BasicDataSource]: Bean property 'maxAge' is not writable or has an invalid setter method. Does the parameter type of the setter match the return type of the getter?
        ... 3 more
Dec 04, 2015 5:41:36 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: Error listenerStart
Dec 04, 2015 5:41:36 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [/DocGemStudioTest] startup failed due to previous errors
2015-12-04 17:41:36,955 [http-bio-8080-exec-63] WARN  lifecycle.ShutdownOperations  - Error occurred running shutdown operation: BeanFactory not initialized or already closed - call 'refresh' before accessing beans via the ApplicationContext
java.lang.IllegalStateException: BeanFactory not initialized or already closed - call 'refresh' before accessing beans via the ApplicationContext
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Dec 04, 2015 5:41:36 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/DocGemStudioTest] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

EDIT 4:

I've edited the properties block and now application starts. I need to check if error occurs after few hours.

 properties{
                 minIdle = 5
                 maxIdle = 25
                 maxWait = 10000

                 timeBetweenEvictionRunsMillis = 5000
                 minEvictableIdleTimeMillis = 60000
                 validationQuery = "SELECT 1"
                 validationQueryTimeout = 3

                 testOnBorrow = true
                 testWhileIdle = true
                 testOnReturn = false

                 defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
        }
Nembutal answered 28/7, 2015 at 13:1 Comment(10)
Did you purposely leave in the "urlofapp/myappname?" in your dataSource url to hide it for this question? Or is this a cut and paste issue?Myeloid
I only use "urlofapp/myappname" instead of real names only for the question, obviouslyNembutal
Does your mysql instance accepts connections from remote server? did you change the default configurations?Stylograph
Yes, the connection works correctly. I see the exception after some hours after last connection...in the above exception it is like 38 hoursNembutal
Here is a link to what sounds like the same issue: #23103247Myeloid
dspano thanks. Now I've added maxAge="180000" testOnBorrow="true" testWhileIdle="true" and we will see what happensNembutal
add &autoReconnect=true or &autoReconnect=yes to your DB connection URLRye
it seems not working too...I've added &autoReconnect=true but it does not workingNembutal
Is your Grails app and MySQL server are on two different machines?Archaeornis
no they are on the same server machineNembutal
R
1

So I believe everything other than dbCreate, url, username, and password needs to be in a properties block:

dataSource {
    dbCreate = "update"
    url = "jdbc:mysql://urlofapp/myappname?useUnicode=yes&characterEncoding=UTF-8&autoReconnect=true"
    username = "root"
    password = "root"
    properties {
        minIdle = 5
        maxIdle = 25
        maxWait = 10000
        maxAge = 10 * 60000
        timeBetweenEvictionRunsMillis = 5000
        minEvictableIdleTimeMillis = 60000
        validationQuery = "SELECT 1"
        validationQueryTimeout = 3
        validationInterval = 15000
        testOnBorrow = true
        testWhileIdle = true
        testOnReturn = false
        jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
        defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
    }
}
Retrograde answered 3/12, 2015 at 14:49 Comment(6)
I try and I will let you know.Nembutal
Adding properties block, application does not startNembutal
This should be the proper answer. Do you get any error message when it does not start? The validateionQuery are the properties that does the trick to keep the session alive.Jadwigajae
You can also change the interactive_timeout setting in MySql but you probably don't want to go down that road. It is better to have a validationQuery that runs every now and then to keep the connection alive.Jadwigajae
@Jadwigajae I've posted the stacktrace. The only thing that I've noticed is about maxAge...what could be the problem?Nembutal
@Nembutal remove the maxAge propertyRetrograde
A
1

Although, this should be strange but if the Grails app and the MySQL server both are on the same machine then use localhost or 127.0.0. instead of real IP address xxx.xxx.xxx.xxx

dataSource {
   dbCreate = "update"
   url = "jdbc:mysql://localhost:3306/docgem_test_db?useUnicode=yes&characterEncoding=UTF-8&autoReconnect=true"
   // Your other properties
}
Archaeornis answered 8/12, 2015 at 11:27 Comment(1)
@Nembutal I didn't realize at first that you the author of the question itself trying to remove your IP from the post but later I accepted your edit. But I just want to warn you that your real IP is still visible in the edit history of my answer so consider adding more security if your IP has your production app or try to change it if possible. (I'll delete this comment as well).Archaeornis
J
0

Please try the below not the addition of driverClassName and dialect and I also added pooled in case you need that

dataSource {
    pooled = true
    driverClassName = "com.mysql.jdbc.Driver"
    dialect = "org.hibernate.dialect.MySQL5InnoDBDialect"
    dbCreate = "update"
    url = "jdbc:mysql://urlofapp/myappname? useUnicode=yes&characterEncoding=UTF-8&autoReconnect=true"
    username = "root"
    password = "root"
    properties {
        minIdle = 5
        maxIdle = 25
        maxWait = 10000
        maxAge = 10 * 60000
        timeBetweenEvictionRunsMillis = 5000
        minEvictableIdleTimeMillis = 60000
        validationQuery = "SELECT 1"
        validationQueryTimeout = 3
        validationInterval = 15000
        testOnBorrow = true
        testWhileIdle = true
        testOnReturn = false
        jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
       defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}

}

Johiah answered 10/12, 2015 at 12:11 Comment(0)
S
-2

Looks to me like MySQL is timing out it's connection to you. Probably not much you can do about that.

Can you consider closing the connection if there's no traffic on it for a while and then re-open it when it's needed again?

Sinotibetan answered 19/11, 2015 at 17:11 Comment(2)
What do you mean with "you"?Do you mean that it is not possible to solve it in another way?Nembutal
I'm saying that it looks like the timeout is happening on the MySQL side of the connection. Unless you control the configuration of MySQL, there may be little you can do about it other that change your approach to how you are handling the connection. This is essentially the same answer as stenix gave you on Dec 4 '15.Sinotibetan

© 2022 - 2024 — McMap. All rights reserved.