BufferedReader doesn't read all lines from file

Question

I am trying to read /proc/net/xt_qtaguid/stats in Android 6.
Using cat command, I get this:

2 a0 0 0 123456 311 48329 737 48 1
3 b0 0 0 0 0 0 0 0
4 c0 123456 311 48329 737 48 1
5 d0 111111 111 22222 222 33 1

My java code tries to read the file line by line:

File sysDataFile = new File(PATH_TO_FILE);
BufferedReader bufReader = null;
FileReader fileReader = null;
try {
   fileReader = new FileReader(sysDataFile);
   bufReader = new BufferedReader(fileReader);

   String line = null;
   while ((line = bufferedReader.readLine()) != null) {
       // print to console each line
       System.out.println("Line: " + line);
   }
 } catch (IOException e) {
    System.out.println("IOException thrown!");           
 } finally {
   bufReader.close();
   fileReader.close();
 }

When I run the above code, it only print out the first two lines in console:

Line: 2 a0 0 0 123456 311 48329 737 48 1
Line: 3 b0 0 0 0 0 0 0 0

Why?

Falconiform

Yes, same file for sure, I am thinking may be 2nd line ends with some hidden special character that stoped the code continue reading, but I am not sure, that's why I ask here. – Falconiform

Try xxd command to display file in hexa. details here – Push

@B.Bri, and then??? – Falconiform

@Falconiform open the file in some text editor(nano for example), set the cursor at the beggining of 3rd line and then press backspace. Press 'Enter' then to create a normal new line character. It usually works... – Lillis

@AndreyCheboksarov, the idea of my program is to read the file programmatically, manually editing the file is not a solution for me. I am seeking for a way to programmatically solve this problem. – Falconiform

Your posted code references bufferedReader but it's undefined. Are you sure you posted the correct code? Are you sure that only the first 2 lines are printed with your real code? If yes, then please upload your input file to somewhere we can download and inspect it. Do not copy paste, because maybe it's not accurate. We need the actual file to debug this, not something copy-pasted. – Foretopsail

BufferedReader: docs.oracle.com/javase/7/docs/api/java/io/BufferedReader.html – Falconiform

Are you trying to read a file under /proc/ ? Those are special, with special permissions and size reported as 0, which might trip Java's readers (see this question, although it does not have very good answers)... I have not managed to reproduce it, even with an old Java 7. Maybe it would help if you could specify what file exactly is this, your kernel version, and java version. – Hepplewhite

I am trying to read /proc/net/xt_qtaguid/stats in Android 6. – Falconiform

seems related https://mcmap.net/q/2029529/-why-i-can-39-t-read-proc-net-xt_qtaguid-stats-correctly-by-filereader-in-android-ics – Talus

Have you tried to change IOException in the catch block to Throwable. Perhaps some other exception type is been throw. Although typically you would see the corresponding stack trace in the common java command line program, that could be silenced some how so is always good generalize the code here to check for that explicitly. – Forethought

Does it bomb in the same character/offset every time? Your problem is not lines it is characters. – Junkie

EOL characters could be the problem , if you can open the file through console cat - e filename and check the content – Leftwich

Answer 1

I am trying to read /proc/net/xt_qtaguid/stats in Android 6.

In Linux, files under /proc/ are not actually files: they are handled by procfs, a special filesystem that executes code each time the file entry is read or written (code being a callback function defined in a kernel module). So those pseudo-files are not static like regular files, but completely dynamic. The size gives an interesting clue: (most of) those files have a length of 0 (as can be seen with ls -l), but when read they show some content.

In short, it is to be expected that reading that same file from 2 different contexts yields 2 different results.

In this instance, the "file" callbacks are handled by xt_qtaguid module for Android, which manages "per-application/delegated data usage monitoring".

This answer says:

this virtual file's read_proc function limit the uid, every application can only read the header and its own line.

The first part is a bit vague but seems to indicate the difference is based on user id, and the module will only "print" 2 lines of data when a regular application reads this file (please note Android assigns a unique user ID to each application and runs it as that user in a separate process).

You don't give enough details but I have to assume that when you print it from adb using cat, you probably don't have the same user id and permissions as when you try to read it from your application. I did not track the exact implementation details (if you want to, the source for this module can be read here), but other variables might come into play.

The doc says:

In the case of applications that provide network data transfer as a service, such as the download manager, media streaming service, etc, it is possible to attribute the ownership of the network data transfer to the UID of the requesting application using the TrafficStats.setThreadStatsUid() function call. The caller must hold the android.permission.MODIFY_NETWORK_ACCOUNTING permission to re-assign the ownership of the network traffic.

So a process/application can use TrafficStats.setThreadStatsUid() in order to get more lines from that file, but that requires MODIFY_NETWORK_ACCOUNTING permission.

Hepplewhite

1. The file size is irrelevant. 2. You can run cat inside or outside adb, and there is no reason for the userID to be different. 3. Anything cat can do, Java can do. Ultimately both call read(). What the kernel does internally is up to the kernel. It doesn't care whether it is cat or java calling read(). – Hautrhin

File size does not explain the difference, but I think it is very relevant to indicate the special nature of the file, maybe that was unclear, so I edited. I also made the rest more vague because I don't have a definite answer about the particulars of that specific file. The first line in OP's question was not there initially, it was much less clear earlier. I asked if they were reading /proc, and when that was confirmed I thought an explanation about procfs would be useful. – Hepplewhite

@EJP Also, I thought that in Android, each app runs with its own user id, so that would explain different user IDs. "The Android system assigns a unique user ID (UID) to each Android application and runs it as that user in a separate process." – Hepplewhite

Answer 2

Since performance is not an issue, the easiest and safest option would be to launch cat in a process from java redirect the standard output to an stream and read the contents in a reader. Using this as guidance: Run cat command in Android

No need to modify permissions to provide a backtick

Junkie

Answer 3

The readLine() method reads until the next '\n' character. Hypothetically, if you have two consecutive '\n' characters, then readLine() could return null prematurely. However, if this was the case, then cat should have put an extra line in between those two lines.

At this point you can try two things:

1. check the charset your system file and your program use are the same

2. change this:

   String line = null;
   while ((line = bufferedReader.readLine()) != null) {
       // print to console each line
       System.out.println("Line: " + line);
   }
   

   to this:

   char c = null;
   while ((c = (char)bufferedReader.read()) != null) {
       // print to console each char
       System.out.println("Line: " + c);
   }
   

and see what happens.

Bron

nevermind, the question skadya has linked to in your original comment thread might just be your problem. – Bron

readLine() reads until the next line terminator. Two of them in a row will produce a blank line. "". Not a premature null. BufferedReader.read() returns int, which cannot be null, and neither can a char. Answer does not make sense, and code does not compile. – Hautrhin

But isn't new line ('\n') a line terminator? I'm actually curious. – Bron

Yes, and so is \r\n, and so is \r by itself. See the Javadoc. – Hautrhin