Do you own a Debenu Quick PDF Library version 7, 8, 9, 10, 11, 12, 13 or iSEDQuickPDF license? Upgrade to Debenu Quick PDF Library 14 today!
Textextraction: Determine the used codepage |
Post Reply |
Author | |
Ingo
Moderator Group Joined: 29 Oct 05 Status: Offline Points: 3524 |
Post Options
Thanks(0)
Posted: 22 Dec 11 at 3:36PM |
Hi!
Now myself ... ;-)
I'm an extensive user of the extract functionality still i'm mostly working with version 7.26 together with Delphi 2007 and Delphi 5.
Actually i'm trying to get the textcontents as unicode.
I can extract the text made in arabic, russian and many other languages as long as they were created with utf8 and after extraction i'll get it all with ...
ansistring := utf8decode(widestring)
As samples i've many foreign pdf-documents. There are few russian documents, too.
Few i can extract 'cause they were made with utf8 - other russian documents failed 'cause they were made with codepage 1251.
My questions now:
Is there a functionality available to detect the used codepage in a textcontent in an automated way?
How to decode codepage 1251 - something similar to utf8decode?
Hope someone can help me out.
Thanks a lot in advance.
Cheers, Merry Christmas and a Happy New Year to all of you
Ingo
Edited by Ingo - 22 Dec 11 at 3:39PM |
|
edvoigt
Senior Member Joined: 26 Mar 11 Location: Berlin, Germany Status: Offline Points: 111 |
Post Options
Thanks(0)
|
Hi Ingo,
the PDF-Spec gives no Results if I search for "codepage". Therefore it is not so simple. But there is an explanation in 5.5.5 Character Encoding. I hope this is the key. To verify this, it is necessary to have a longer look in an inflated encoding dictionary and an entry in it. For a such test a PDF with codepage 1251 is needed. This means a longer way thru some objects. Can you mail me a such PDF? Erläuterungen natürlich besser für mich auf deutsch. Cheers, Werner |
|
Ingo
Moderator Group Joined: 29 Oct 05 Status: Offline Points: 3524 |
Post Options
Thanks(0)
|
Hi Werner!
You've got an email ... right now ;-) Thanks in advance. Cheers, Ingo Edited by Ingo - 22 Dec 11 at 10:32PM |
|
AndrewC
Moderator Group Joined: 08 Dec 10 Location: Geelong, Aust Status: Offline Points: 841 |
Post Options
Thanks(0)
|
Extracting text is not an easy process. Each font can either be defined as single byte or multi byte font and they also contain an Encoding entry in the CMap which is similar to a code page. To make things more complex a font can also contain a ToUnicode mapping array to help with text extraction routines. On top of that a font can contain a Differences array which can remap any character code to a new code.
Also some documents contains subsetted fonts which allow you remap any character code to any other code and this can make text extraction impossible as you can tell the font to draw a 'A' but the font actually draws as a 'B'. QPL hides this functionality to make text extraction easy to use. GetPageText options 3 - 8 are more advanced than option 0 and 1. The only way to understand what is going on is to look at the CMaps contained in each font. Andrew
|
|
Ingo
Moderator Group Joined: 29 Oct 05 Status: Offline Points: 3524 |
Post Options
Thanks(0)
|
Hi Werner, Hi Andrew!
I begin to realize ... really not an easy job ;-)
I have to read ... Thank you both for the links.
Cheers, Ingo
|
|
edvoigt
Senior Member Joined: 26 Mar 11 Location: Berlin, Germany Status: Offline Points: 111 |
Post Options
Thanks(0)
|
Hi Ingo, Hi Andrew,
the goal is not to paint a glyph correctly, therefore it is enough to figure out the use of the codepage 1251 with its charset. Here a quick&dirty solution without to much work: function IsQP1251(fn: string): boolean; var QP: TQuickPDF; obj: string; i, n, glyphno, error, p: integer; begin Result := false; QP := TQuickPDF.Create; if QP.UnlockKey({$I PDFkey.inc}) = 1 // 8.xx then begin QP.LoadFromFile(fn, ''); n := QP.GetObjectCount; i := 1; repeat // search for an PDF-object /Encoding with afii-codes for cyrillic inside obj := QP.GetObjectToString(i); // afii10017-afii10846 if (Pos('/Encoding', obj)>0) and (Pos('/Differences', obj)>0) then begin p := Pos('afii10', obj); if (p>0) then begin // 17..846 Val(copy(obj, p+6, 3), glyphno, error); Result := (error=0) and (glyphno>=17) and (glyphno<=84 6); end; end; inc(i); until (i>n) or Result; end; end; This works under Delphi7 with QuickPDF 8.13 The basic idea is: inside the PDF is an encoding-object, which may start so: /Type /Encoding /BaseEncoding /WinAnsiEncoding /Differences So I go over all objects and look for /Encoding and /Differences. The /Differences-array is the key. There are strings starting with "afii" followed by a number. They denote a special interpretation of the value of a byte. It is dirty, because I am not sure about is there ever a such /Encoding-entry in the PDF. It is dirty, because I go the short way directly to an object. Better to go along through the tree (/Procset => /Font => /Encoding), but the afii-codes are rather unique in conjunction with /Differences. I dont know, may this Encoding-object be compressed too? In this case it would be more work, another reason for my wish (case 9605) for a GetInflatedObjectTo... as great brother of GetObjectTo... used in the solution above. An overview you may get here http://partners.adobe.com/public/developer/en/opentype/aglfn13.txt A good source is http://www.adobe.com/content/dam/Adobe/en/devnet/font/pdfs/5013.Cyrillic_Font_Spec.pdf too. The solution works with the two examples I got from Ingo. Maybe there is more to do. But it is a first entry. Cheers and merry christmas Werner Edited by edvoigt - 23 Dec 11 at 12:42PM |
|
Ingo
Moderator Group Joined: 29 Oct 05 Status: Offline Points: 3524 |
Post Options
Thanks(0)
|
Hi Werner!
Thanks a lot for this! Now i can go further ... So there's the question which are the most used worldwide encode-formats ;-) To you and all the other ones here
a Merry Christmas and a Happy New Year,
Ingo
|
|
edvoigt
Senior Member Joined: 26 Mar 11 Location: Berlin, Germany Status: Offline Points: 111 |
Post Options
Thanks(0)
|
Hi,
it seems to be a good source here: http://www.science.co.il/language/locale-codes.asp?s=codepage There are some other forwarding links. And you need the relation between afii-codes and copepage. May it helps. Werner |
|
edvoigt
Senior Member Joined: 26 Mar 11 Location: Berlin, Germany Status: Offline Points: 111 |
Post Options
Thanks(0)
|
Hi,
here a short adwise how make it not so dirty and more sure. The solution above determes if a font is using afii-codes. Consequently you have to figure out, in which part of the PDF-Text is which font used. The test only if a such font exists is rather unexact. It may happen, there is a mixture of languages (and codepages) on one page. This is rather sure if there are parts from different PDFs combined to a new one. Therefore there is a journey first through the fontdefinitions and than through the content needed. Werner |
|
Post Reply | |
Tweet
|
Forum Jump | Forum Permissions You cannot post new topics in this forum You cannot reply to topics in this forum You cannot delete your posts in this forum You cannot edit your posts in this forum You cannot create polls in this forum You cannot vote in polls in this forum |
Copyright © 2017 Debenu. Debenu Quick PDF Library is a PDF SDK. All rights reserved. About — Contact — Blog — Support — Online Store