Extract columns of text from a pdf file using iText

Question

I need to extract text from pdf files using iText.

The problem is: some pdf files contain 2 columns and when I extract text I get a text file where columns are merged as the result (i.e. text from both columns in the same line)

this is the code:

public class pdf
{
    private static String INPUTFILE = "http://www.revuemedecinetropicale.com/TAP_519-522_-_AO_07151GT_Rasoamananjara__ao.pdf" ;
    private static String OUTPUTFILE = "c:/new3.pdf";

    public static void main(String[] args) throws DocumentException, IOException {
        Document document = new Document();
        PdfWriter writer = PdfWriter.getInstance(document, new FileOutputStream(OUTPUTFILE));
        document.open();

        PdfReader reader = new PdfReader(INPUTFILE);
        int n = reader.getNumberOfPages();

        PdfImportedPage page;

        // Go through all pages
        for (int i = 1; i <= n; i++) {
            page = writer.getImportedPage(reader, i);
            Image instance = Image.getInstance(page);
            document.add(instance);
        }

        document.close();

        PdfReader readerN = new PdfReader(OUTPUTFILE);
        for (int i = 1; i <= n; i++) {
            String myLine = PdfTextExtractor.getTextFromPage(readerN,i);
            System.out.println(myLine);

            try {             
                FileWriter fw = new FileWriter("c:/yo.txt",true);
                fw.write(myLine);
                fw.close();
            }catch (IOException ioe) {ioe.printStackTrace(); }
    }
}

Could you please help me with the task?

Saltus

Answer 1

I am the author of the iText text extraction sub-system. What you need to do is develop your own text extraction strategy (if you look at how PdfTextExtractor.getTextFromPage is implemented, you will see that you can provide a pluggable strategy).

How you are going to determine where columns start and stop is entirely up to you - this is a difficult problem - PDF doesn't have any concept of columns (heck, it doesn't even have a concept of words - just putting together the text extraction that the default strategy provides is quite tricky). If you know in advanced where the columns are, then you can use a region filter on the text render listener callback (there is code in the iText library for doing this, and the latest version of the iText In Action book gives a detailed example).

If you need to obtain columns from arbitrary data, you've got some algorithm work ahead of you (if you get something working, I'd love to take a look). Some ideas on how to approach this:

1. Use an algorithm similar to that used in the default text extraction strategy (LocationAware...) to obtain a list of words and X/Y locations (be sure to account for rotation angle as well)
2. For each word, draw an imaginary line running the full height of the page. Scan for all other words that start at the same X position.
3. While scanning, also look for words that intersect the X position (but do not start on the X position). This will give you potential location for column start/stop Y positions on the page.
4. Once you have column X and Y, you can resort to a region filtered approach

Another approach that may be equally feasible would be to analyze draw operations and look for long horizontal and vertical lines (assuming the columns are demarcated in a table-like format). Right now, the iText content parser doesn't have callbacks for these operations, but it would be possible to add them without major difficulty.

Kakalina

Thanks @kevin How do PDF readers do this so well? – Headmost

@david004 I doubt very much that the PDF Readers do an amazing job on all PDF files. It would be possible to be more clever with the separator character that we insert between words (which is what I suspect the PDF reader apps are doing). Basically, if the distance between a character and the next is greater than some threshold (maybe twice the width of a space character?) use a tab character instead of space. This may be good enough for a lot of scenarios - but definitely not good enough for general purpose parsing. If it would help, I could look at adding tab insertion behavior... – Kakalina

Is there alghoritm with C# example to count and restream (extract to db) words from specific row in .PDF table? – Viewy

Answer 2

Tables do not exist as structures in PDF unless the file uses Structured content. Do you understand what a PDF file is? I wrote a blog article explaining the issues of text extraction at http://www.jpedal.org/PDFblog/?p=228

Peddada

Answer 3

You could also try PdfBox, but it all goes back to lack of structure in the PDF - its primarily an end file output format for display.

Peddada

Answer 4

PDFTextStream is the one! At least I am able to identify the column values. Earlier, I was using iText and got stuck in defining strategy. Its hard.

This api separates column cells by putting more spaces. Its fixed. you can put logic. (this was missing in iText).

import com.snowtide.PDF;
import com.snowtide.pdf.Document;
import com.snowtide.pdf.OutputTarget;

public class PDFText {
    public static void main(String[] args) throws java.io.IOException {
        String pdfFilePath = "xyz.pdf";

        Document pdf = PDF.open(pdfFilePath);
        StringBuilder text = new StringBuilder(1024);
        pdf.pipe(new OutputTarget(text));
        pdf.close();
        System.out.println(text);
   }
}

Question has been asked related to this on stackoverflow!

Stiletto

The OP wrote "I need to extract text from pdf files using iText." So how does your answer help him do that? (Saying PDFTextStream is built upon itext doesn't count. ) – Kuhlmann

It is a promotion of a paid product – Thibeault

Answer 5

The file you are extracting from is pretty complex for data extraction purposes. There are tables, images, multiple, columns. You will need special algorithms to determine the reading order and also process the table data.

What are you trying to achieve here ? You could use a commercial OCR engine instead and let it do all the hard work and then process the data from there.

Callicrates

Answer 6

I know my answer is a bit late. But I'm using the following code to read certain pages from PDF files. I didn't have any problem reading columns, no merged text, each column is being printed aside from the other.

    /**
 * Get plain text from a specific page in a pdf file.
 * @param pdfPath
 * @return
 * @throws IOException
 */
public static String getPageContent(String pdfPath, int pageNumber) throws IOException
{
    PdfReader reader = new PdfReader(pdfPath); 

    StringWriter output = new StringWriter();  

        try {
            output.append(PdfTextExtractor.getTextFromPage(reader, pageNumber, new SimpleTextExtractionStrategy()));

        } catch (OutOfMemoryError e) {

            // TODO Auto-generated catch block
            e.printStackTrace();
        }

    return output.toString();
}

If you are looking into extracting part of a page, let's say 1 column only, then you need to get the dimensions of the column. It's still a bit tricky but you might be able to figure this out if you already knew the begining text of the column (in a way to estimate the width and height). This can be done by using a rectangular area. See code below, and sorry if I got the point measurement wrong. In the code below I try to get the whole page dimension.

public static String getPageContent(String pdfPath, int pageNumber) throws IOException
{

    PDDocument pdDoc = PDDocument.load(pdfPath);
    PDPage specPage = (PDPage)pdDoc.getDocumentCatalog().getAllPages().get( 0 );

PDFTextStripperByArea stripper = new PDFTextStripperByArea();
stripper.setSortByPosition( true );
float width = (specPage.getMediaBox().getHeight())*25.4f;
float height = (specPage.getMediaBox().getWidth())*25.4f;
Rectangle rect = new Rectangle( 0, 0, Math.round(width), Math.round(height));
stripper.addRegion( "class1", rect );
List allPages = pdDoc.getDocumentCatalog().getAllPages();
PDPage firstPage = (PDPage)allPages.get( pageNumber-1 );
stripper.extractRegions( firstPage );

return stripper.getTextForRegion( "class1" );

}

Tullusus