PHP: Read from certain point in file

Question

Similar to: How to read only 5 last line of the text file in PHP?

I have a large log file and I want to be able to show 100 lines from position X in the file. I need to use fseek rather than file() because the log file is too large.

I have a similar function but it will only read from the end of the file. How can it be modified so that a start position can be specified as well? I would also need to start at the end of the file.

function read_line($filename, $lines, $revers = false)
{
    $offset = -1;
    $i = 0;
    $fp = @fopen($filename, "r");
    while( $lines && fseek($fp, $offset, SEEK_END) >= 0 ) {
        $c = fgetc($fp);
        if($c == "\n" || $c == "\r"){
            $lines--;
            if($revers){
                $read[$i] = strrev($read[$i]);
                $i++;
            }
        }
        if($revers) $read[$i] .= $c;
        else $read .= $c;
        $offset--;
    }
    fclose ($fp);
    if($revers){
        if($read[$i] == "\n" || $read[$i] == "\r")
            array_pop($read);
        else $read[$i] = strrev($read[$i]);
        return implode('',$read);
    }
    return strrev(rtrim($read,"\n\r"));
}

What I'm trying to do is create a web based log viewer that will start from the end of the file and display 100 lines, and when pressing the "Next" button, the next 100 lines preceding it will be shown.

Janettejaneva

Here are yet more ways to do this in order of efficiency: unix.stackexchange.com/questions/94318/… – Janettejaneva

Answer 1

This uses fseek to read 100 lines of a file starting from a specified offset. If the offset is greater than the number of lines in the log, the first 100 lines are read.

In your application, you could pass the current offset through the query string for prev and next and base the next offset on that. You could also store and pass the current file position for more efficiency.

<?php

$GLOBALS["interval"] = 100;

read_log();

function read_log()
{
   $fp = fopen("log", "r");
   $offset = determine_offset();
   $interval = $GLOBALS["interval"];
   if (seek_to_offset($fp, $offset) != -1)
   {
      show_next_button($offset, $interval);
   }
   $lines = array();
   for ($ii = 0; $ii < $interval; $ii++)
   {
      $lines[] = trim(fgets($fp));
   }
   echo "<pre>";
   print_r(array_reverse($lines));
}

// Get the offset from the query string or default to the interval
function determine_offset()
{
   $interval = $GLOBALS["interval"];
   if (isset($_GET["offset"]))
   {
      return intval($_GET["offset"]) + $interval;
   }
   return $interval;
}

function show_next_button($offset, $interval)
{
   $next_offset = $offset + $interval;
   echo "<a href=\"?offset=" . $offset . "\">Next</a>";
}

// Seek to the end of the file, then seek backward $offset lines
function seek_to_offset($fp, $offset)
{
   fseek($fp, 0, SEEK_END);
   for ($ii = 0; $ii < $offset; $ii++)
   {
      if (seek_to_previous_line($fp) == -1)
      {
         rewind($fp);
         return -1;
      }
   }
}

// Seek backward by char until line break
function seek_to_previous_line($fp)
{
   fseek($fp, -2, SEEK_CUR);
   while (fgetc($fp) != "\n")
   {
      if (fseek($fp, -2, SEEK_CUR) == -1)
      {
         return -1;
      }
   }
}

Chard

If the number of lines in the file is not known and I want to start viewing from the end of the file, how would I use your code? – Janettejaneva

nevermind, the linux command wc -l my_log.log will output line numbers – Janettejaneva

I'm not sure you'll need that. If you pass an offset larger than the # of lines, this just shows the 100 lines at the beginning of the log. fseek returns -1 when it tries to seek past the beginning. I'll add a comment where it hits the beginning of the file. – Chard

In general or at the end of the log? When it detects the end, you could disable or remove the next button. For normal cases, you would pass the current offset via the query string (i.e. viewlog.php?offset=400) and add 100 to that. I'll add this to the code. – Chard

Thank you! I actually didn't realize what $GLOBALS["interval"] was meaning, but I perfectly understand now. Your code is perfect! Thanks a bunch! Cheers :) – Janettejaneva

OK cool, glad it's working :D -- I added the code for the next link anyway just in case. – Chard

Any particular reason why we are not using file_get_contents ($filename $use_include_path, $context, $offset, $maxlen) – Often

Answer 2

If you're on Unix, you can utilize the sed tool. For example: to get line 10-20 from a file:

 sed -n 10,20p errors.log

And you can do this in your script:

<?php
$page  = 1;
$limit = 100;
$off   = ($page * $limit) - ($limit - 1);

exec("sed -n $off,".($limit+$off-1)."p errors.log", $out);
print_r($out);

The lines are available in $out array.

Demoss

This works great, but I would want to be able to read from the end of the log file, and the number of lines in the file is not known – Janettejaneva

nevermind, the linux command wc -l my_log.log will output this. Cheers! – Janettejaneva

Answer 3

Is "position X" measured in lines or bytes? If lines, you can easily use SplFileObject to seek to a certain line and then read 100 lines:

$file = new SplFileObject('log.txt');
$file->seek(199); // go to line 200

for($i = 0; $i < 100 and $file->valid(); $i++, $file->next())
{
    echo $file->current();
}

If position X is measured in bytes, isn't it a simple matter of changing your initial $offset = -1 to a different value?

Cady

This nice but I want to read from the end of the file first. But the length of the file is not known. – Janettejaneva

The linux command wc -l my_log.log will print the # of line numbers – Janettejaneva

Answer 4

I would do it as followed:

function readFileFunc($tempFile){
    if(@!file_exists($tempFile)){
        return FALSE;
    }else{
        return file($tempFile);
    }
}
$textArray = readFileFunc('./data/yourTextfile.txt');
$slicePos = count($textArray)-101;
if($slicePos < 0){
    $slicePos = 0;  
}
$last100 = array_slice($textArray, $slicePos);
$last100 = implode('<br />', $last100);
echo $last100;

Cordalia

I'm not sure if PHP regex function can handle large text. – Demoss

Sure it CAN!!!!! And at least it is much faster than loops. I have a recent project with a text file of 48.1MB with more than 500'000 lines and it works much faster than looping or while through such thing. but up to you! Learn Regular Expressions and you will see programming differently regular-expressions.info/reference.html – Cordalia

As the user is already having memory issues saving the file contents to a variable for analysis, while definitely faster, will still take up a lot of memory on a very large file. – Typescript

He could cache those pieces and serve it further or put it into the user session but to call fseek every time will just raise other disadvantages. Although, both are possible solutions. I thought he just doesn't wanna serve the browser with more than 100 lines, what you don't do with my code, all server side... but maybe he has server memory issues, who knows? Although, it sounds that this concerns the browser output... – Cordalia

And php is very fast on server side text handling. We could talk about resource issues on image processing though... – Cordalia

If you are going to read the whole file, might as well use file() and pick out the lines from the resulting array. No regex needed. – Servais