This is what worked for me:
Assuming you have your selectors set in the drawable res folder (like Xingang Huang showed above).
In your MainActivity (where you setup your TabLayout) you include your array of icon selectors and then you loop through it like this:
for (int i = 0; i < yourTabLayout.getTabCount(); i++) {
ImageView imageView = new ImageView(this); //your context, in this case MainActivity.class
imageView.setImageResource(arr_tabIcons[i]); //tabIcons is the array of icons
if (i==0) {
imageView.setSelected(true);
}
yourTabLayout.getTabAt(i).setCustomView(imageView);
}
tab.setIcon(R.drawable.icon)
works as well but in my case the icons looked really small, so I had to use the solution with the ImageView to fill the tab view.
Happy coding ;)