Packet loss while receiving UDP broadcast in android device

Question

For receiving UDP broadcast packets from the server to an android device, i used a service class and listen for packets in a thread. It receives the packet successfully. The problem is that if multiple packets are being sent from the server in the same time then packet loss will be the result.

I even tried with a queue and processing the received packets in separate thread then also i am not getting the packet. I am completely new to network programming any help would be widely appreciated

void startListenForUdpBroadcast() {
        UDPBroadcastThread = new Thread(new Runnable() {
            public void run() {
                try {
                    InetAddress broadcastIP = InetAddress.getByName(UdpConstants.IP_ADDRESS);
                    Integer port = UdpConstants.RECEIVER_PORT;
                    while (shouldRestartSocketListen) {
                        listenAndWaitAndThrowIntent(broadcastIP, port);
                    }

                } catch (Exception e) {
                    Log.i("UDP", "no longer listening for UDP broadcasts cause of error " + e.getMessage());
                }
            }
        });
        UDPBroadcastThread.setPriority(Thread.MAX_PRIORITY); //Setting The Listener thread to MAX PRIORITY to minimize packet loss.
        UDPBroadcastThread.start();
    }

This code listens for new packets and pushes to queue

private void listenAndWaitAndThrowIntent(InetAddress broadcastIP, Integer port) throws Exception {
        byte[] recvBuf = new byte[64000];
        if (socket == null || socket.isClosed()) {
            socket = new DatagramSocket(port, broadcastIP);
            socket.setBroadcast(true);
        }
//socket.setSoTimeout(1000);
        DatagramPacket packet = new DatagramPacket(recvBuf, recvBuf.length);

        socket.receive(packet);
        messQueue.add(packet);

    }

This checks the queue for new messages and process it

 /**
     * @purpose Checking queue and If anything is added to the queue then broadcast it to UI
     */
    private void checkQueue() {
        queueThread = new Thread(new Runnable() {
            public void run() {
                try {

                    while (shouldRestartSocketListen) {
                        if (!messQueue.isEmpty()) {
                            broadcastIntent(messQueue.poll());
                        }
                    }

                } catch (Exception e) {

                }
            }
        });
        queueThread.start();
    }

Saltworks

What server do you have in your system? is it another android device? Is the communication done using WIFI? Does your program works in a LAN IP network environment or does it goes to the web? – Thun

@Thun a server implemented in C#. Yes comminication is done via WiFi. Its works only in LAN IP. – Saltworks

Answer 1

The problem with UDP is that your sender (your server) does not know you (your android device) missed some. It's not lost because you can't read it fast enough, sometimes just over the air interference/congestion or a busy socket.

The receiver would only know if:

1. you get an error while processing data since you're missing data
2. OR your UDP packets are numbered sequentially in its header and you detect a missing number (eg. 1,2,4 - missing 3)

Once the packet is lost, it's lost. You got two options:

1. implement a resend request: upon detection of a missing packet, the receiver would need to notify the sender to resend that missing packet until it does get it, and your packet processing might be halted until it does
2. OR be able to ignore it, "hey, we can do it without him", and fill-in with blank data (eg. a bitmap would have some blank pixels, like a broken image)
3. throttle your sending speed down so the packets wont jam up and get lost
4. the smaller your packets, the more likely they'll live

(option 1: all this resend requesting is just pseudo-TCP, so you might just consider abandoning UDP and go TCP)

Nusku

so there is no problem with my listener? or can i do anything more to make it a little bit optimized – Saltworks

resend packet request if lost seems to be the only solution – Saltworks

Looks ok, as long as theres nothing potentially blocking in the listening part – Nusku

All good points - I would add is that, depending on how quickly the server sends datagrams back-to-back, packets certainly can be lost in the kernel of the android device. In addition, mobile/wireless networks are notoriously spotty - but given that you appear to see losses only when sending a burst from the server, it sounds more like queue overflow somewhere in the network - perhaps at the android device - but also possibly in a router queue along the way. You could also try pacing the datagrams you send from the server (i.e. wait a short time between each send). – Spoil

@Spoil The packet loss is not only when the intervals are short sometime eventhough they are slow its lost – Saltworks

@Nusku After running the application for long time the listener just stops is there any idea why? – Saltworks

When the device hits an idle threshold, connections such as these will be shutdown. You will need to implement a wakelock to keep it alive. – Nusku

@Nusku Ok i will acquire a wakelock in onResume and release it in onPause and give it a try. Earlier i just added the FLAG_KEEP_SCREEN_ON flag. – Saltworks

Answer 2

I think your problem is mainly that you use Udp Broadcast over wifi.

Their are two very well documented answers why this is a very slow way to operate and why there are much more packet losts:

• answer number one.

• answer number two.

The thing I did to solve the extremely slow bandwidth was some kind of multi-unicast protocol:

1. Manage the list of clients you have connected.
2. Send each packet you have in your server to all of your clients separately with send call.

This is the code in java:

  DatagramPacket packet = new DatagramPacket(buffer, size);
  packet.setPort(PORT);

  for (byte[] clientAddress : ClientsAddressList) {
        packet.setAddress(InetAddress.getByAddress(clientAddress));
        transceiverSocket.send(packet);
  }

Thun

Answer 3

If you receive multiple datagrams in a short burst, your receiver loop may have trouble keeping up, and the OS-level RCVBUF in the socket may overflow (causing the OS to drop a packet it indeed did receive).

You might get better handling of short bursts if you increase the RCVBUF size. Prior to doing this, get an idea of how big it is already via socket.getReceiveBufferSize. Also bear in mind that the number of bytes in the receive buffer must accommodate not just the payload but also the headers and the sk_buff structure that stores packets in the kernel (see, e.g. lxr.free-electrons.com/source/include/linux/…).

You can adjust the recieve buffer size via socket.setReceiveBufferSize - but bear in mind that this message is just a hint, and may be overridden by settings in the kernel (if you request a size bigger than the max size allowable by the current kernel network settings, then you'll only get the max size).

After requesting a bigger receive buffer, you should double check what the kernel has allowed by calling socket.getReceiveBufferSize.

If you have the right permissions, you should be able to tweak the max buffer size the kernel will allow - see e.g. https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html

[ Note that, in general, this will accommodate for short bursts where datagrams arrive faster than your client loop can read them - but if the client loop is chronically slower than datagram delivery, you'll still get occasional drops due to overflow. In this case, you need to speed up your client loop, or slow down the sender.

Also, as otherwise noted in other answers, packets may actually be dropped by your network - and mobile networks especially may be prone to this - so if you absolutely need guaranteed delivery you should use TCP. However, if this were your primary problem, you would expect to see dropped packets even when your server sends them slowly, rather than in a burst.]

Spoil

each packets contains a 10 letter word. I will give setReceiveBufferSize by calling getReceiveBufferSize a try – Saltworks

yes sometime its dropped eventhough its been send slowly from the server. In the answer you mentioned about speeding up the client loop to match the packet delivery speed. Is there any way to achieve this. I tried to increase the BufferSize but i guess like you said even though its large kernal internally changes it to its max limit i guess. – Saltworks

can we clear this buffer manually? because when the application runs for long time this listener is not working – Saltworks

The buffer is generally cleared by rapidly reading datagrams out of it. If you are seeing loss at long intervals too (e.g 10ms is generally quite a long time, network wise), then it could be other parts of the network that are dropping packets. [ e.g., I believe many wireless access points keep long queues, and re-try packets when there is wireless interference. If so, then they're also dropping packets when that queue fills ] Given your symptoms, I might switch to building more robustness in the face of packet loss - switch to TCP, or have your client request re-sends of missing packets. – Spoil

Answer 4

I suppose that you are capturing only a single packet by saying

socket.receive(packet);

This is a Blocking I/O call which will wait infinitely until it receives a packet so once first packet arrives it is done waiting and next command executes i.e

messQueue.add(packet);

However when multiple packets are been received you need to continue receiving packets. in your case you just stopped receiving packages after arrival of first package

Note: UDP being a un-reliable protocol doesn't guarantee packet delivery so there might be a case a packet is lost , However this can't be a problem on every run of your program , However a nice way to check whether the packet is hitting your machine and problem is within your application (application is not able to handle the packets recieved) use tcpdump (it's a command-line utility for linux-based OS or MAC) use the following command

 sudo tcpdump -i <interface name(one that handles traffic) eth0, eth1 .. en0, en1 (for MAC)> host <ipaddress or hostname of UDP Broadcast server>

Example:

sudo tcpdump -i en1 host 187.156.13.5

(if tcpdump command not found then go forward and install it)

By using this command you will see packets pumping in from destination ump server on terminal if you see more then one packets arriving then you would be sure that packets are arriving at machine , However application falls short to address the packet

it might help

Donny

socket.receive(packet) is written in a thread inside a service will it block I/O . I tried the command line for tcp dump in my mac but i got error. Packets are coming to my android device , running tcpdump in my mac will it trace the packets? – Saltworks

tcpdump will work on your Mac but for your android device it won't , yes even if you run it on a separate thread the thread will go into Blocking IO state (Since thread goes into blocking IO state and not your main programme which has spawn the thread so it would appear that the you are making a Non Blocking IO call in term of Main Programme but thread is went under blocking mode). – Donny

However your problem looks you just need to capture the second packet even if it is on thread because thread is also like a programme but a smaller one which runs in background. For a start just try and add this two lines and try to test whether you have the second packet or not (this should work) DatagramPacket packet1 = new DatagramPacket(recvBuf, recvBuf.length); socket.receive(packet1); Now this two lines added will look for second packet as well , This is not a proper way(as for each new packet you add two more lines) but still i would help to identify the problem properly. – Donny

And if the above thing works use infinite while loop to accept packet this accept another packet as soon as it is done receiving one while(true){ //code to get packet and process it } – Donny

Answer 5

With reference to above explanation by me following changes you can make to make code behave according to requirement

I suppose you can make following changes to make your problem code work instead of creating socket into listenAndWaitAndThrowIntent(InetAddress broadcastIP, Integer port ) method create it in startListenForUdpBroadcast() as follows

socket = new DatagramSocket(port, broadcastIP);
socket.setBroadcast(true);
while (shouldRestartSocketListen) {
     listenAndWaitAndThrowIntent(broadcastIP, port, socket);
}

Now you also need to change implementation of listenAndWaitAndThrowIntent method as follows

private void listenAndWaitAndThrowIntent(InetAddress broadcastIP, 
Integer port, DatagramSocket socket) throws Exception {
    byte[] recvBuf = new byte[64000];
    //socket.setSoTimeout(1000);
    //change value of condition in for loop to desired number of packets you would like to receive in below example it is 2 so it captures two packets
    for (int i=0 ; i <= 2 ; i++ ){
    DatagramPacket packet = new DatagramPacket(recvBuf, recvBuf.length);

    socket.receive(packet);
    messQueue.add(packet);
    }

}

Try this it should work !! it might help

Donny

you can refer this docs.oracle.com/javase/tutorial/networking/datagrams/examples/… piece of code – Donny

This is almost certainly not what you want to do. This code creates a new socket with each listen, and waits for a total of two packets with each listen. When the old packet is torn down, you will lose all packets that may have been queued to it. – Spoil

I would have kept socket outside the while . My mistake thanks for clearing @Spoil – Donny