Systems such as databases, but also for example queueing systems that you use through JMS have the concept of transactions. A transaction is treated as a unit of work; at the end of doing the work, such as inserting, updating or deleting records in the database, you commit the transaction and then the database definitively does the work; or you rollback the transaction and then everything done in the transaction is cancelled.
In some cases, your software has to perform operations over multiple different systems. For example, you might need to insert data into multiple databases, or insert something in the database and put a message on a queue.
If you want to do such combinations of operations as if they are in one transaction, then you need a distributed transaction system - a system that can combine the transactions of the different systems into one. That way you can write your code as if it's running inside a single transaction; the distributed transaction system automatically commits or rolls back the transactions in the underlying systems.
To make it more concrete: Suppose that you insert a record in a database, and put a message on a queue, and you want to do this inside one transaction. When something goes wrong with putting the message on the queue, you also want the database transaction to be rolled back, so that you don't have a record in the database but not a corresponding message on the queue. Instead of manually keeping track of the transactions of the database and the queue system (including handling all combinations of possible errors), you can use a distributed transaction system.
XA is a standard for working with distributed transactions. You can work with XA transactions in Java through the Java Transaction API (JTA). Java EE servers have support for this built-in. If you're not using a Java EE server, then you can use a separate library that implements JTA such as Narayana or Atomikos.
Each method in DAO open new connection to database and close it in the end of the method.
Normally this isn't how you should write DAOs. Opening a database connection is a relatively slow operation; if you open a new database connection for every method that you call in a DAO, your program is most like going to run slowly. You should at least use a connection pool, that manages a number of database connections and allows you to reuse already open connections.
If I use one instance of database, and each method in DAO write to different table, do I have to use XA Datasource? since transaction occured in service layer but only in one instance database
If your DAO methods each open their own connection, then they will run in separate transactions. Whether this is a problem or not depends on what your application needs to do. An XA datasource is not the solution to make them run in one transaction. Instead, you should both let them use the same connection and transaction.
XA transactions are really only useful if you are using multiple database systems or other systems, and you want to be able to perform transactions that span across these systems.
