Loading and displaying large text files

Question

In a Swing application, I sometimes need to support read-only access to large, line-oriented text files that are slow to load: logs, dumps, traces, etc. For small amounts of data, a suitable Document and JTextComponent are fine, as shown here. I understand the human limitations of browsing large amounts of data, but the problematic stuff seems like it's always in the biggest file. Is there any practical alternative for larger amounts of text in the 10-100 megabyte, million-line range?

Babylonia

You should think of providing the user a different experience when an extremely long file is selected. Consider letting the user download the file; go to a special "search" interface that returns information surrounding search terms, or some other motif. Think of what the user wants or needs to do before trying to provide some "view a million line" file the same way you vie a 100 line file. – Hauptmann

@ErstwhileIII: Yes, the GUI equivalent of grep … | more; please consider posing a related answer. – Babylonia

+1 there is a direct correlation (latency, performance) betweens document size and to display stylled text, word and line wrap, epsecially custom XxxEditorKit isn't designated for large text without paginations, put FileIO to List<List<String/Document>> notifier should be max value for/from JProgressBar – Bipinnate

@mKorbel: Good point; the text component classes support editing of rich content; the JTable approach suggested below supports line-oriented access to plain text. – Babylonia

@Babylonia not sure if is JTable (maybe better could be JList) designated for, not sure how is handled the caching for painting off_screen (in the case that structure is tabular) and to provide better latency, more performance than plain JTextArea (monospaced font) without word and line wrap, maybe I'm wrong but I think that for JTextComponents probably some optimization missed, if any is there implemented, – Bipinnate

Answer 1

I would separate the problem.

The first one is model - Document building speed

The second is the Document rendering - building tree of views to represent the Document.

A question is whether you need font effects like keywords colorizing?

I would start from Document building part. IMHO reading the file via EditorKit.read() should be fast even for big files. I would use the PainDocument for the purpose and check whether the pure model is built fast enough for your application. If yes it's fine just use the Document as model. If not implement your own Document interface because AbstractDocument has plenty of methods for update processing (e.g. writeLock).

When we have the Document loading fast enough we have to solve the Document rendering. BY default the views used in javax.swing.text are really flexible. They are designed as base classes to be extended - thus has a lot of code we don't need. E.g. measuring.

For the feature I would use Monospaced font, we don't need wrap so measurements of the view widht is fast = longest row char count * char widht.

The height is also char height * amount of lines.

So our PLainTextViewReplacement is really fast. Also we don't have to render the whole view but just a fragment visible in our scroll pane. Thus rendering could be mmuch much faster.

Of course there should be a lot of work to provide correct caret navigation, selection etc.

Diamonddiamondback

Thank you for looking at this; I'd neglected to consider AbstractDocument , but PlainDocument would probably be OK; I don't think I'd need Font.MONOSPACED, but fixed height would allow flyweight rendering. – Babylonia

@Babylonia I would start from plainDocumentInstance.replace(0,0,theLongText,null) to check the doc creation speed. If it's good enough you can try to replace PlainView – Diamonddiamondback

Accepted for your helpful reminder about read(), tested here. – Babylonia

Answer 2

Because of the size, you'll surely want to load the file in the background to avoid blocking the event dispatch thread; SwingWorker is a common choice. Instead of using a Document, consider updating a TableModel and displaying the lines of text in the rows of a JTable. This offers several advantages:

• Results will begin appearing immediately, and there will be reduced perceived latency.

• JTable uses the flyweight pattern for rendering, which scales well into the multi-megabyte, million-line range.

• You can parse the input as it is being read to create an arbitrary column structure.

• You can leverage the sorting and filtering features of JTable, for example.

• You can use TablePopupEditor to focus on a single line.

Addendum: The example below uses DefaultTableModel for convenience. To reduce overhead, extend AbstractTableModel and manage a List<String> or List<RowData>, as shown here. The example displays indeterminate progress; changes to display intermediate progress are shown here.

Code:

import java.awt.BorderLayout;
import java.awt.EventQueue;
import java.beans.PropertyChangeEvent;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.util.List;
import javax.swing.JFrame;
import javax.swing.JProgressBar;
import javax.swing.JScrollPane;
import javax.swing.JTable;
import javax.swing.SwingWorker;
import javax.swing.table.DefaultTableModel;
import javax.swing.table.TableModel;

/**
 * @see https://mcmap.net/q/541767/-loading-and-displaying-large-text-files
 */
public class DisplayLog {

    private static final String NAME = "/var/log/install.log";

    private static class LogWorker extends SwingWorker<TableModel, String> {

        private final File file;
        private final DefaultTableModel model;

        private LogWorker(File file, DefaultTableModel model) {
            this.file = file;
            this.model = model;
            model.setColumnIdentifiers(new Object[]{file.getAbsolutePath()});
        }

        @Override
        protected TableModel doInBackground() throws Exception {
            BufferedReader br = new BufferedReader(new FileReader(file));
            String s;
            while ((s = br.readLine()) != null) {
                publish(s);
            }
            return model;
        }

        @Override
        protected void process(List<String> chunks) {
            for (String s : chunks) {
                model.addRow(new Object[]{s});
            }
        }
    }

    private void display() {
        JFrame f = new JFrame("DisplayLog");
        f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        DefaultTableModel model = new DefaultTableModel();
        JTable table = new JTable(model);
        JProgressBar jpb = new JProgressBar();
        f.add(jpb, BorderLayout.NORTH);
        f.add(new JScrollPane(table));
        f.pack();
        f.setLocationRelativeTo(null);
        f.setVisible(true);
        LogWorker lw = new LogWorker(new File(NAME), model);
        lw.addPropertyChangeListener((PropertyChangeEvent e) -> {
            SwingWorker.StateValue s = (SwingWorker.StateValue) e.getNewValue();
            jpb.setIndeterminate(s.equals(SwingWorker.StateValue.STARTED));
        });
        lw.execute();
    }

    public static void main(String[] args) {
        EventQueue.invokeLater(() -> {
            new DisplayLog().display();
        });
    }
}

Babylonia

tell us more about filtering and sorting.. how can i find a certain word within that large text file? – Miramontes

@MartinFrank As linked in the answer, check out How to use tables, Sorting and Filtering for more details and examples... – Nester

@Martin Frank Oracle tutorial How to use Tables - part Sorting and Filtering, or matcher for JTextComponent (again with working code example), all methods are done out of GUI, bz default in the JComponents model, sure in most cases this model can repaced with standard arrays, anything as is a List, Vector or Document (model for JTextComponents) – Bipinnate

@MartinFrank: It's a broad topic, but I added a link to my take on mKorbel's related question. – Babylonia

I loaded a 95 MB file, which started as a 357 MB process. On a profiler, there were String, char[] and Vectors of nearly the same count. I wonder how the call to model.addRow could be optimized suiting Swing. Collections.addAll directly to Vector had no effect. Also as I scroll back and forth, the memory usage shoots unconditionally to 400 MB+, But must say the scroll was smooth as silk. – Barr

@Rajeev: Extend AbstractTableModel and manage a List<String> directly, for example. – Babylonia

Answer 3

As I was struggeling with a similar use case I implemented a simple paging solution. It is far from perfect but works maybe someone finds it helpful.

In combination with a jtextarea it works ok but with a JEditorPane the performance is miserable.

If someone comes up with a better solution I would like to know about.

package net.ifao.tools.arcticrequester.gui.panel;


import java.awt.Adjustable;
import java.awt.event.AdjustmentEvent;
import java.awt.event.AdjustmentListener;
import java.io.BufferedReader;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintStream;
import java.io.StringReader;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingDeque;

import javax.swing.JEditorPane;
import javax.swing.JScrollPane;
import javax.swing.text.Document;
import javax.swing.text.EditorKit;
import javax.swing.text.JTextComponent;


/**
 * A class that manages the visibility of file content visualized with a textarea within a scrollbar.
 * Approx. 2050 lines from the file are visible at a time. Data is loaded from a file and
 * displayed while the user is scrolling. The chunks are loaded dynamically.
 *
 * @author dostricki
 *
 */
public class VisibilityManager
   implements AdjustmentListener
{

   private int lastLoadedLineFrom;
   private int lastLoadedLineTo;
   private int numberOfLines = 0;
   private File file;

   private boolean enabled = false;
   private boolean showLines = false;

   // load 1000 lines before the first visible line
   // and 1000 lines after the last vissible line
   private static final int LOAD_LINES_BEFORE_AND_AFTER_VIEWPORT = 1000;

   // margin until when no load is triggered.
   // moving the viewport more then 900 lines up or down should trigger a reload
   private static final int VIEWPORT_LINES_MOVE_THRASHOLD = 900;


   private JScrollPane scrollPane;
   private JTextComponent textComponent;

   private final BlockingQueue<Adjustable> queue;

   public VisibilityManager(JScrollPane scrollPane, JTextComponent textArea)
   {
      this.scrollPane = scrollPane;
      this.textComponent = textArea;
      queue = new LinkedBlockingDeque<>();
      startConsumer();

      scrollPane.getVerticalScrollBar().addAdjustmentListener(this);
   }

   private void startConsumer()
   {
      Thread scrollEventConsumer = new Thread()
      {

         @Override
         public void run()
         {
            while (true) {
               try {

                  // if multiple events occured just process one
                  queue.take();
                  if (!queue.isEmpty()) {
                     List<Adjustable> events = new ArrayList<>();
                     queue.drainTo(events);
                     //System.out.println("Handling scroll event. " + events.size() + " queued events dropped");
                  }

                  doHandleScrollEvent();

               }
               catch (InterruptedException e) {
                  e.printStackTrace();
               }
            }
         }
      };
      scrollEventConsumer.start();
   }

   public void setFile(File file)
   {
      this.file = file;

      try {
         this.numberOfLines = countNumberOfLines(file);
      }
      catch (IOException e1) {
         e1.printStackTrace();
      }
      int showLineMax = Math.min(getNumberOfLines(), 100);

      // show the first chunk immediately
      showLinesBuffererdReader(1, showLineMax, 0);

      this.enabled = true;
   }

   /**
    * precalculates the number of lines in the document - necessary
    * to replace the correct amount of preceeding and following
    * lines with EOL's so that the height of the scrollpane does never change.
    *
    * @param file
    * @return
    * @throws IOException
    */
   private int countNumberOfLines(File file)
      throws IOException
   {

      int numberOfLines = 0;

      //@formatter:off
      try (BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(file),StandardCharsets.UTF_8));) {
         while (reader.ready()) {
            reader.readLine();
            ++numberOfLines;
         }
      }
      //@formatter:on

      return numberOfLines;
   }


   /****************************************
    *                Getter
    ****************************************/


   public int getNumberOfLines()
   {
      return numberOfLines;
   }

   public int getNumberOfLinesBuffer()
   {
      return LOAD_LINES_BEFORE_AND_AFTER_VIEWPORT;
   }

   public boolean isEnabled()
   {
      return enabled;
   }

   /****************************************
    *              Setter
    ****************************************/


   public void setLastLoadedLines(int lineFrom, int lineTo)
   {
      this.lastLoadedLineFrom = lineFrom;
      this.lastLoadedLineTo = lineTo;
   }

   public void setEnabled(boolean enabled)
   {
      this.enabled = enabled;
   }

   public void setShowLines(boolean showLines)
   {
      this.showLines = showLines;
   }


   /****************************************
    *           Calculation
    ****************************************/


   private boolean needsUpdate(int fromLine, int toLine)
   {
      boolean isBefore = fromLine < (this.lastLoadedLineFrom - VIEWPORT_LINES_MOVE_THRASHOLD);
      boolean isAfter = toLine > (this.lastLoadedLineTo + VIEWPORT_LINES_MOVE_THRASHOLD);

      if (isBefore || isAfter) {
         return true;
      } else {
         return false;
      }
   }

   private void showLinesBuffererdReader(int from, int to, int firstLineVisible)
   {
      //load also the buffer lines before
      from = from - getNumberOfLinesBuffer();
      //make sure it's valid
      from = Math.max(1, from);

      // load also the buffer lines after
      to = to + getNumberOfLinesBuffer();
      //make sure it's valid
      to = Math.min(getNumberOfLines(), to);

      FileChannel fileChannel = null;
      try (BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(file)))) {

         StringBuffer content = new StringBuffer();
         int newCaretPosition = 0;

         // fill leading empty lines
         for (long i = 1; i < from; ++i) {
            if (i == firstLineVisible) {
               newCaretPosition = content.length() + 1;
            }
            if (showLines) {
               content.append(i).append(": ");
            }
            content.append('\n');
         }

         // read/write lines with content
         int j = 0;
         while (reader.ready() && j <= to) {
            ++j;
            String line = reader.readLine();
            if (j >= from && j <= to) {
               if (j == firstLineVisible) {
                  newCaretPosition = content.length() + 1;
               }
               if (showLines) {
                  content.append(j).append(": ");
               }
               content.append(line).append('\n');
            }
         }

         // fill trailing empty lines
         for (int i = to + 1; i <= getNumberOfLines(); ++i) {
            if (i == firstLineVisible) {
               newCaretPosition = content.length() + 1;
            }
            if (showLines) {
               content.append(i).append(": ");
            }
            content.append('\n');
         }

         updateTextInUI(content);

         // workaround for page up/down - it changes the caret position
         // so we are re-setting it to the first visible line
         // scrolling by scrollbars does not change the caret
         //textComponent.setCaretPosition(newCaretPosition);
      }
      catch (IOException e) {
         e.printStackTrace();
      }
      finally {

         try {
            if (fileChannel != null) {
               fileChannel.close();
            }
         }
         catch (IOException e) {
            e.printStackTrace();
         }
      }
   }

   /**
    * @param content
    * @throws IOException
    */
   private void updateTextInUI(StringBuffer content)
      throws IOException
   {
      if (textComponent instanceof JEditorPane) {
         JEditorPane edit = ((JEditorPane) textComponent);
         EditorKit editorKit = edit.getEditorKit();
         Document createDefaultDocument = editorKit.createDefaultDocument();
         createDefaultDocument.putProperty("IgnoreCharsetDirective", Boolean.TRUE);
         try {
            editorKit.read(new StringReader(content.toString()), createDefaultDocument, 0);
         }
         catch (Exception e) {
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            e.printStackTrace(new PrintStream(out));
            edit.setText(new String(out.toByteArray()));
         }
         edit.setDocument(createDefaultDocument);
      } else {
         textComponent.setText(content.toString());
      }
   }


   /****************************************
    *           Eventing
    ****************************************/


   /**
    * fired when scrolling happens in any of the cases and ways.
    * Events are cached through a queue so that simultanious events
    * don't trigger unnecessary update actions
    * @see java.awt.event.AdjustmentListener#adjustmentValueChanged(java.awt.event.AdjustmentEvent)
    */
   @Override
   public void adjustmentValueChanged(AdjustmentEvent evt)
   {
      Adjustable source = evt.getAdjustable();
      if (evt.getValueIsAdjusting()) {
         return;
      }

      if (source != null) {
         try {
            queue.put(source);
         }
         catch (InterruptedException e) {
            e.printStackTrace();
         }
      }
   }


   private void doHandleScrollEvent()
   {
      // determine which lines to request to be loaded into the
      int height = this.scrollPane.getVerticalScrollBar().getMaximum();
      int lines = getNumberOfLines();
      if (lines == 0) {
         return;
      }
      float heightPerLine = height / lines;
      int visibleLines = Math.round(this.scrollPane.getVerticalScrollBar().getVisibleAmount() / heightPerLine);
      int firstLineVisible = (int) Math.ceil(this.scrollPane.getVerticalScrollBar().getValue() / heightPerLine);

      int fromLine = Math.max(firstLineVisible, 1);
      if (fromLine > lines) {
         fromLine = lines;
      }

      int toLine = Math.min(firstLineVisible + visibleLines, lines);
      if (needsUpdate(fromLine, toLine)) {
         if (enabled) {
            setLastLoadedLines(fromLine, toLine);
            showLinesBuffererdReader(fromLine, toLine, firstLineVisible);
         }
      }
   }
}

usage:

import java.awt.BorderLayout;
import java.awt.EventQueue;
import java.awt.event.MouseEvent;
import java.awt.event.MouseListener;
import java.io.File;

import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.JScrollPane;
import javax.swing.JTextArea;
import javax.swing.UIManager;
import javax.swing.UnsupportedLookAndFeelException;
import javax.swing.border.EmptyBorder;
import javax.swing.text.DefaultCaret;

import net.ifao.tools.arcticrequester.gui.panel.VisibilityManager;


public class TestFrame
   extends JFrame
   implements MouseListener
{

   private VisibilityManager visibilityManager;


   public static void main(String[] args)
   {
      EventQueue.invokeLater(new Runnable()
      {
         @Override
         public void run()
         {
            try {
               TestFrame frame = new TestFrame();
               frame.setVisible(true);
            }
            catch (Exception e) {
               e.printStackTrace();
            }
         }
      });
   }

   /**
    * Create the frame.
    */
   public TestFrame()
   {
      try {
         UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
      }
      catch (ClassNotFoundException | InstantiationException | IllegalAccessException | UnsupportedLookAndFeelException e2) {
         // TODO Auto-generated catch block
         e2.printStackTrace();
      }

      setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
      setBounds(100, 100, 650, 500);

      JPanel contentPane = new JPanel();
      contentPane.setBorder(new EmptyBorder(5, 5, 5, 5));
      setContentPane(contentPane);
      contentPane.setLayout(new BorderLayout(0, 0));

      JTextArea textArea = new JTextArea();
      textArea.setEditable(false);
      textArea.addMouseListener(this);
      textArea.setAutoscrolls(false);

      textArea.setCaretPosition(0);
      DefaultCaret caret = (DefaultCaret) textArea.getCaret();
      caret.setUpdatePolicy(DefaultCaret.NEVER_UPDATE);

      JScrollPane scrollPane = new JScrollPane(textArea);

      contentPane.add(scrollPane);

      visibilityManager = new VisibilityManager(scrollPane, textArea);

      visibilityManager.setShowLines(true);
      File file = new File("C:/iFAO/workspaces/polaris2/git/requester/ArcticRequester/src/test/java/responseview_20200603.tmp");
      visibilityManager.setFile(file);


      this.dispose();
   }


   /**
    * @see java.awt.event.MouseListener#mouseClicked(java.awt.event.MouseEvent)
    */
   @Override
   public void mouseClicked(MouseEvent e)
   {
      boolean doScroll = !visibilityManager.isEnabled();
      this.visibilityManager.setEnabled(doScroll);
      System.out.println("scrolling set to " + doScroll);
   }

   /**
    * @see java.awt.event.MouseListener#mousePressed(java.awt.event.MouseEvent)
    */
   @Override
   public void mousePressed(MouseEvent e)
   {
      // TODO Auto-generated method stub

   }

   /**
    * @see java.awt.event.MouseListener#mouseReleased(java.awt.event.MouseEvent)
    */
   @Override
   public void mouseReleased(MouseEvent e)
   {
      // TODO Auto-generated method stub

   }

   /**
    * @see java.awt.event.MouseListener#mouseEntered(java.awt.event.MouseEvent)
    */
   @Override
   public void mouseEntered(MouseEvent e)
   {
      // TODO Auto-generated method stub

   }

   /**
    * @see java.awt.event.MouseListener#mouseExited(java.awt.event.MouseEvent)
    */
   @Override
   public void mouseExited(MouseEvent e)
   {
      // TODO Auto-generated method stub

   }

}

Ruderal