Table empty cells text extraction

Message Topic Search Topic Options Post Reply Create New Topic Printable Version Translate Topic

   Hi,I have a PDF with a big (multi page) table and I'm trying to extract all text present in that table, using the QPDF library for Delphi 2007 and ExtractFilePageText function (with Options "8" as the best algorithm choice for that type of table).

The problem is: not all table cells are filled with text , some are empty, so I'm going crazy searching a way to let the library put in the output text file some sort of placeholder to inform me that some cells are empty. Now I have an output text file with a new line of text for every table cell with text into, but NO lines at all when a cell contains no text (so no way to know that in a specific line and column there is an empty cell, a type of information that I need to recostruct the whole table content).

My ideal solution could be to get the library to output a blank line when it finds an empty cell, and not, I repeat, simply skip to next cell with text (letting me with the only chance to reconstruct the total number of cells empty for each line of the table , but no chance to know in which column I have text and in which not).

Any help will be much appreciated!

Marco

Author	Message Topic Search Topic Options Post Reply Create New Topic Printable Version Translate Topic
MarcoCir Members Profile Find Members Posts Beginner Joined: 08 Jun 14 Location: Italia Status: Offline Points: 5	Post Options Post Reply Quote MarcoCir Report Post Thanks(0) Quote Reply Topic: Table empty cells text extraction Posted: 08 Jun 14 at 12:17PM
	Hi, I have a PDF with a big (multi page) table and I'm trying to extract all text present in that table, using the QPDF library for Delphi 2007 and ExtractFilePageText function (with Options "8" as the best algorithm choice for that type of table). The problem is: not all table cells are filled with text , some are empty, so I'm going crazy searching a way to let the library put in the output text file some sort of placeholder to inform me that some cells are empty. Now I have an output text file with a new line of text for every table cell with text into, but NO lines at all when a cell contains no text (so no way to know that in a specific line and column there is an empty cell, a type of information that I need to recostruct the whole table content). My ideal solution could be to get the library to output a blank line when it finds an empty cell, and not, I repeat, simply skip to next cell with text (letting me with the only chance to reconstruct the total number of cells empty for each line of the table , but no chance to know in which column I have text and in which not). Any help will be much appreciated! Marco

Ingo Members Profile Find Members Posts Moderator Group Joined: 29 Oct 05 Status: Offline Points: 3524	Post Options Post Reply Quote Ingo Report Post Thanks(0) Quote Reply Posted: 10 Jun 14 at 9:25PM
	Hi Marco, you should use option 4. Then you'll get returning strings with position data. Having the position data page by page you can check where there's something missing. My idea is getting data for example for row 1 col 1, row 1 col 3, row 1 col 4, row 2 col 1, row 2 col 2, ... let you know that row 1 col 2 is an empty field ;-) Cheers and welcome here, Ingo
	Cheers, Ingo

MarcoCir Members Profile Find Members Posts Beginner Joined: 08 Jun 14 Location: Italia Status: Offline Points: 5	Post Options Post Reply Quote MarcoCir Report Post Thanks(0) Quote Reply Posted: 12 Jun 14 at 10:28AM
	Hi Ingo. I've take a look a bit deeper to the option 4. First of all I think I've found a little problem in the documentation: the coordinates of the points returned in the option 4 are in the Y,X order, not X,Y (moving from a cell of the table to the next one in horizontal order - same row - it seems that are the Ys that are changing instead of, of course, the Xs. That said, yes, did you tell me to look for the coordinates returned cell by cell and from those values, with a simple algorithm, find if there is some cell that is missing from the extraction process? thank you, I'll try this path now! Marco

Ingo Members Profile Find Members Posts Moderator Group Joined: 29 Oct 05 Status: Offline Points: 3524	Post Options Post Reply Quote Ingo Report Post Thanks(0) Quote Reply Posted: 12 Jun 14 at 12:29PM
	Hi Marco, regarding the coordinates this could have to do with the use of rotate-functionality. Rotation will rotate the position data, too ;-) If you want to calculate and using some algos you should Keep in mind that pdf-documents can look similar but can be different inside. For example two documents with format DIN A4 but one was initially landscape and then rotated to DIN A4. So check the rotation first (there's an explained function in the reference) and use SetOrigin. Cheers, Ingo Edited by Ingo - 12 Jun 14 at 12:31PM
	Cheers, Ingo

AndrewC Members Profile Find Members Posts Moderator Group Joined: 08 Dec 10 Location: Geelong, Aust Status: Offline Points: 841	Post Options Post Reply Quote AndrewC Report Post Thanks(0) Quote Reply Posted: 16 Jun 14 at 7:17AM
	Marco, You should call QP.NormalizePage(0); before calling the text extraction functions. A PDF file has no concept of cells, sentences or paragraphs. PDF text is just placed on the page like a jigsaw puzzle. Some pieces can contain 1 character, others multiple characters and there are many functions that make this more complex such as character spacing, word spacing etc.. Debenu Quick PDF Library just like any other PDF library has to collect all the characters and their positions on the page and group characters into words and then into lines. It is not an exact science. Your best option is to try to use GetPageText(7) and the SetTextExtractionScaling function if you are using tightly spaced proportional fonts. If you provide a link to the PDF I could then suggest the best method for extraction. I have a few years of experience in complex table extraction from PDF files. Andrew

MarcoCir Members Profile Find Members Posts Beginner Joined: 08 Jun 14 Location: Italia Status: Offline Points: 5	Post Options Post Reply Quote MarcoCir Report Post Thanks(0) Quote Reply Posted: 16 Jun 14 at 9:21AM
	Hi Andrew, and thanks. This is the link to that PDF file: http://www.inbet.it/pdf/calcioq.pdf As you see, some cells are empty (without odds), so my problems. I've followed the path suggested by Ingo, with some improvement (but the algorithm is not trivial ;) ). Marco

AndrewC Members Profile Find Members Posts Moderator Group Joined: 08 Dec 10 Location: Geelong, Aust Status: Offline Points: 841	Post Options Post Reply Quote AndrewC Report Post Thanks(0) Quote Reply Posted: 16 Jun 14 at 3:07PM
	Marco, There is going to be no easy way to read this with Debenu Quick PDF Library without some complex strategies. How many pages do you need to process and how often ? Andrew.

MarcoCir Members Profile Find Members Posts Beginner Joined: 08 Jun 14 Location: Italia Status: Offline Points: 5	Post Options Post Reply Quote MarcoCir Report Post Thanks(0) Quote Reply Posted: 16 Jun 14 at 3:16PM
	Hi Andrew, my files are not a lot of pages long, but I need to process them ...20 times a day, every day of the year. So, according to Ingo suggestions I've now developed some Delphi code with the option 4, using x and y coordinates to find if a cell is empty (so, skipped by the library) and the results are not so bad. Anyway, I'll continue to search for a better solution, thank you all. Marco

AndrewC Members Profile Find Members Posts Moderator Group Joined: 08 Dec 10 Location: Geelong, Aust Status: Offline Points: 841	Post Options Post Reply Quote AndrewC Report Post Thanks(0) Quote Reply Posted: 16 Jun 14 at 3:17PM
	Marco, Try this QP.SetTextExtractionScaling(0, 2, 8); QP.SetTextExtractionWordGap(0.2); And then call GetPageText(7); You can then split the text into columns and trim leading and trailing spaces. Andrew.