"FATAL: the database system is shutting down" while creating connection to PostgreSQL

Question

I am getting FATAL: the database system is shutting down errors while creating PostgreSQL JDBC connections to a PostgreSQL 9.2 server. The specific exception path I'm getting from JDBC is here:

Caused by: org.postgresql.util.PSQLException: FATAL: the database system is shutting down
    at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173)
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
    at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136)
    at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
    at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
    at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
    at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
    at org.postgresql.Driver.makeConnection(Driver.java:393)
    at org.postgresql.Driver.connect(Driver.java:267)

From various log files (from PostgreSQL, from our management layer, from the app using PostgreSQL), I do not see any database shutdown actually happening (other connections are created to PostgreSQL as usual, no shutdown was initiated from our management layer, etc.), however in the PostgreSQL server logs I do see the error message with the same timestamp:

2014-06-16 12:30:00.736 GMT LOG:  connection received: host=127.0.0.1 port=38530
2014-06-16 12:30:00.737 GMT FATAL:  the database system is shutting down

Researching online, I learned that this error message is used whenever PostgreSQL shuts down connections.

Why would PostgreSQL refuse to give me a new connection? Could this be caused by some sort of resource contention? How can I get more information about the error from PostgreSQL itself?

Bilbe

Look further up the database server log to see why it's being shut down. Did you use pg_ctl stop or something? – Middlebrooks

what's PostgreSQL installed port ??? – Holp

Answer 1

This issue turned out to be caused by a very bad misuse of PostgreSQL where our server had booted two PostgreSQL instances on the same data directory (it had deleted postmaster.pid and used a new port number, so the normal safeguards against this behavior were bypassed) which explains why the logs did not hold any useful information -- they were being overwritten by the PostgreSQL instance which wasn't refusing connections. The actual issue was caused by a complex interaction between the competing PostgreSQL instances, and I hope nobody else runs into this issue as well!

Bilbe

What was the solution that fixed it? – Kailey

can you please, provide the solution here? – Hindermost

The solution in my case was not running two PostgreSQL servers on the same set of files. If you don't manually delete the postmaster.pid file, or are more careful than we were being when you do it, then PostgreSQL is designed to prevent you from doing this. – Bilbe

Answer 2

Maybe there is not enough space on hard drive.

Rai

Answer 3

We encountered this problem when the server was restarted. In this case the reboot did not initiate the connection with PostgreSql. Just check the status: "/etc/init.d/postgresql status" if the return is: "no server running" just start: "/etc/init.d/postgresql start"

Markhor

Answer 4

SOLUTION:

1. brew remove postgresql
2. Temporary rename the lock files: mv /tmp/.s.PGSQL.5432.lock /tmp/BK.s.PGSQL.5432.lockand mv /tmp/.s.PGSQL.5432 /tmp/BK.s.PGSQL.5432
3. brew install postgresql

Enjoy!

On

Not ideal in a deployed environment :) – Spies

Answer 5

find the pid of the postgres process.

ps -ef | grep postgres | grep conf

kill the pid

kill -9 <pid>

This should start the process clean.

Ecotone