The main function of a replica set is to provide data redundancy and high availability for your MongoDB deployment. That is, if the primary node in a replica set went down for any reason, a secondary would immediately take over as the new primary (on average within ~10 seconds). See Replication for more details on this topic.
Official MongoDB drivers are aware of this replica set election event, and would provide automatic reconnection and operation retries to the new primary. From the point of view of the app, nothing happened in the database side.
Another advantage of using a replica set with multiple secondaries is the possibility of zero downtime upgrade/maintenance in a rolling fashion. This can be done by taking one secondary offline, do maintenance on it, then do maintenance on the other secondaries, and finally stepping down the primary and do maintenance on it. Again since official MongoDB drivers are aware of these events, you can technically do maintenance on a live database with very minimal impact and no downtime to the app.
This is a different philosophy vs. a monolithic database server, where there is only one true big server. Although there are certain merits in a monolithic deployment (which is a different discussion again :) ), MongoDB was designed as a fault-tolerant distributed database in mind. One immediate drawback of a single server is that the server must be up 100% at all times, or else the app is disrupted. A replica set was designed so that your app can have 100% uptime without putting pressure on individual servers having to have 100% uptime.
As a bonus, a replica set may be able to provide read scalability by setting the driver to read from secondaries (writes must always go to the primary). Note that there should be a careful design should you want to do secondary reads, as this can potentially interfere with the high availability aspect if abused.
In summary, a replica set can provide:
- High availability & fault tolerance
- No downtime maintenance
- Data redundancy to scale reads
without requiring the hardware to be 100% reliable. This is why a replica set is strongly recommended in a prod deployment.
See Replica Set Deployment Architectures for more detailed replica set deployment considerations.