First of all, you have to implement the peer-to-peer protocol stack:
+--------------------------------------+
| NDEF (NFC Data Exchange Format) |
+--------------------------------------+
| SNEP (Simple NDEF Exchange Protocol) |
+--------------------------------------+
| LLCP (Logical Link Control Protocol) |
+--------------------------------------+
| NFCIP-1 |
+--------------------------------------+
You already seem to have this up and running, as you indicate that you "tried to send NDEF messages from PN532 module to smartphone and conversely and it works fine."
So, next you would need to create an NDEF message with a MIME type record (or even better an NFC Forum External Type record) as its first record. In order to use Android's AAR (Android Application Record) facility to bind the NDEF message to only your app (app is either launched or, if not installed, your app's Play Store page is opened), you could additionally add an AAR to the end of your NDEF message.
An NDEF message consisting of only a MIME type record:
+------+------+------+------+------+--------------------------+
| MB=1 | ME=1 | CF=0 | SR=1 | IL=0 | TNF=2 (MIME type record) |
+------+------+------+------+------+--------------------------+
| TYPE LENGTH=32 (0x20) |
+-------------------------------------------------------------+
| PAYLOAD LENGTH=5 (0x05) |
+-------------------------------------------------------------+
| TYPE="application/vnd.test.com.openapp" |
+-------------------------------------------------------------+
| PAYLOAD="Hello" |
+-------------------------------------------------------------+
As a byte array this would look like
{
/* Header byte */ 0xD2,
/* TYPE LENGTH */ 0x20,
/* PAYLOAD LENGTH */ 0x05,
/* TYPE */ 0x61, 0x70, 0x70, 0x6C, 0x69, 0x63, 0x61, 0x74,
0x69, 0x6F, 0x6E, 0x2F, 0x76, 0x6E, 0x64, 0x2E,
0x74, 0x65, 0x73, 0x74, 0x2E, 0x63, 0x6F, 0x6D,
0x2E, 0x6F, 0x70, 0x65, 0x6E, 0x61, 0x70, 0x70,
/* PAYLOAD */ 0x48, 0x65, 0x6C, 0x6C, 0x6F
}
You can then register your app to be launched using an intent filter like this in your manifest:
<intent-filter>
<action android:name="android.nfc.action.NDEF_DISCOVERED" />
<category android:name="android.intent.category.DEFAULT" />
<data android:mimeType="application/vnd.test.com.openapp" />
</intent-filter>
For the AAR variant, you would append an AAR for your app to that message. For instance, if your app has the package name "com.your.app.package":
+------+------+------+------+------+--------------------------+
| MB=1 | ME=0 | CF=0 | SR=1 | IL=0 | TNF=2 (MIME type record) |
+------+------+------+------+------+--------------------------+
| TYPE LENGTH=32 (0x20) |
+-------------------------------------------------------------+
| PAYLOAD LENGTH=5 (0x05) |
+-------------------------------------------------------------+
| TYPE="application/vnd.test.com.openapp" |
+-------------------------------------------------------------+
| PAYLOAD="Hello" |
+-------------------------------------------------------------+
+------+------+------+------+------+--------------------------+
| MB=0 | ME=1 | CF=0 | SR=1 | IL=0 | TNF=4 (External type) |
+------+------+------+------+------+--------------------------+
| TYPE LENGTH=15 (0x0F) |
+-------------------------------------------------------------+
| PAYLOAD LENGTH=20 (0x14) |
+-------------------------------------------------------------+
| TYPE="android.com:pkg" |
+-------------------------------------------------------------+
| PAYLOAD="com.your.app.package" |
+-------------------------------------------------------------+
As a byte array this would look like
{
/* MIME type record */
/* Header byte */ 0x92,
/* TYPE LENGTH */ 0x20,
/* PAYLOAD LENGTH */ 0x05,
/* TYPE */ 0x61, 0x70, 0x70, 0x6C, 0x69, 0x63, 0x61, 0x74,
0x69, 0x6F, 0x6E, 0x2F, 0x76, 0x6E, 0x64, 0x2E,
0x74, 0x65, 0x73, 0x74, 0x2E, 0x63, 0x6F, 0x6D,
0x2E, 0x6F, 0x70, 0x65, 0x6E, 0x61, 0x70, 0x70,
/* PAYLOAD */ 0x48, 0x65, 0x6C, 0x6C, 0x6F,
/* Android Application Record */
/* Header byte */ 0x54,
/* TYPE LENGTH */ 0x0F,
/* PAYLOAD LENGTH */ 0x14,
/* TYPE */ 0x61, 0x6E, 0x64, 0x72, 0x6F, 0x69, 0x64, 0x2E,
0x63, 0x6F, 0x6D, 0x3A, 0x70, 0x6B, 0x67
/* PAYLOAD */ 0x63, 0x6F, 0x6D, 0x2E, 0x79, 0x6F, 0x75, 0x72,
0x2E, 0x61, 0x70, 0x70, 0x2E, 0x70, 0x61, 0x63,
0x6B, 0x61, 0x67, 0x65
}