I need help - I can help - Textextraction: Determine the used codepage

Print Page | Close Window

Textextraction: Determine the used codepage

Printed From: Debenu Quick PDF Library - PDF SDK Community Forum
Category: For Users of the Library
Forum Name: I need help - I can help
Forum Description: Problems and solutions while programming with the Debenu Quick PDF Library and Debenu PDF Viewer SDK
URL: http://www.quickpdf.org/forum/forum_posts.asp?TID=2078
Printed Date: 01 Jul 24 at 5:19AM
Software Version: Web Wiz Forums 11.01 - http://www.webwizforums.com

Topic: Textextraction: Determine the used codepage

Posted By: Ingo
Subject: Textextraction: Determine the used codepage
Date Posted: 22 Dec 11 at 3:36PM

Hi!

Now myself ... ;-)

I'm an extensive user of the extract functionality still i'm mostly working with version 7.26 together with Delphi 2007 and Delphi 5.

Actually i'm trying to get the textcontents as unicode.

I can extract the text made in arabic, russian and many other languages as long as they were created with utf8 and after extraction i'll get it all with ...

ansistring := utf8decode(widestring)

As samples i've many foreign pdf-documents. There are few russian documents, too.

Few i can extract 'cause they were made with utf8 - other russian documents failed 'cause they were made with codepage 1251.

My questions now:

Is there a functionality available to detect the used codepage in a textcontent in an automated way?

How to decode codepage 1251 - something similar to utf8decode?

Hope someone can help me out.

Thanks a lot in advance.

Cheers, Merry Christmas and a Happy New Year to all of you

Ingo

Replies:

Posted By: edvoigt
Date Posted: 22 Dec 11 at 10:19PM

Hi Ingo,

the PDF-Spec gives no Results if I search for "codepage". Therefore it is not so simple. But there is an explanation in 5.5.5 Character Encoding. I hope this is the key.

To verify this, it is necessary to have a longer look in an inflated encoding dictionary and an entry in it. For a such test a PDF with codepage 1251 is needed.

This means a longer way thru some objects. Can you mail me a such PDF?

Erläuterungen natürlich besser für mich auf deutsch.

Cheers,

Werner

Posted By: Ingo
Date Posted: 22 Dec 11 at 10:31PM

Hi Werner!

You've got an email ... right now ;-)
Thanks in advance.

Cheers, Ingo

Posted By: AndrewC
Date Posted: 23 Dec 11 at 7:59AM

Extracting text is not an easy process. Each font can either be defined as single byte or multi byte font and they also contain an Encoding entry in the CMap which is similar to a code page. To make things more complex a font can also contain a ToUnicode mapping array to help with text extraction routines. On top of that a font can contain a Differences array which can remap any character code to a new code.

Also some documents contains subsetted fonts which allow you remap any character code to any other code and this can make text extraction impossible as you can tell the font to draw a 'A' but the font actually draws as a 'B'.

QPL hides this functionality to make text extraction easy to use. GetPageText options 3 - 8 are more advanced than option 0 and 1.

The only way to understand what is going on is to look at the CMaps contained in each font.

http://www.adobe.com/content/dam/Adobe/en/devnet/font/pdfs/5014.CIDFont_Spec.pdf - http://www.adobe.com/content/dam/Adobe/en/devnet/font/pdfs/5014.CIDFont_Spec.pdf

Andrew

Posted By: Ingo
Date Posted: 23 Dec 11 at 8:13AM

Hi Werner, Hi Andrew!

I begin to realize ... really not an easy job ;-)

I have to read ... Thank you both for the links.

Cheers, Ingo

Posted By: edvoigt
Date Posted: 23 Dec 11 at 11:22AM

Hi Ingo, Hi Andrew,

the goal is not to paint a glyph correctly, therefore it is enough to figure out the use of the codepage 1251 with its charset.

Here a quick&dirty solution without to much work:

function IsQP1251(fn: string): boolean;
var
QP: TQuickPDF;
obj: string;
i, n, glyphno, error, p: integer;
begin
Result := false;
QP := TQuickPDF.Create;
if QP.UnlockKey({$I PDFkey.inc}) = 1 // 8.xx
then begin
    QP.LoadFromFile(fn, '');
    n := QP.GetObjectCount;
    i := 1;
    repeat    // search for an PDF-object /Encoding with afii-codes for cyrillic inside
      obj := QP.GetObjectToString(i); // afii10017-afii10846
      if (Pos('/Encoding', obj)>0) and (Pos('/Differences', obj)>0)
      then begin
        p := Pos('afii10', obj);
      if (p>0)
        then begin       // 17..846
          Val(copy(obj, p+6, 3), glyphno, error);
        Result := (error=0) and (glyphno>=17) and (glyphno<=84 6);
        end;
      end;
      inc(i);
    until (i>n) or Result;
end;
end;

This works under Delphi7 with QuickPDF 8.13

The basic idea is:

inside the PDF is an encoding-object, which may start so:
/Type /Encoding /BaseEncoding /WinAnsiEncoding /Differences
So I go over all objects and look for /Encoding and /Differences.

The /Differences-array is the key. There are strings starting with "afii" followed by a number. They denote a special interpretation of the value of a byte.

It is dirty, because I am not sure about is there ever a such /Encoding-entry in the PDF.

It is dirty, because I go the short way directly to an object. Better to go along through the tree (/Procset => /Font => /Encoding), but the afii-codes are rather unique in conjunction with /Differences.

I dont know, may this Encoding-object be compressed too? In this case it would be more work, another reason for my wish (case 9605) for a GetInflatedObjectTo... as great brother of GetObjectTo... used in the solution above.

An overview you may get here
http://partners.adobe.com/public/developer/en/opentype/aglfn13.txt - http://partners.adobe.com/public/developer/en/opentype/aglfn13.txt

A good source is
http://www.adobe.com/content/dam/Adobe/en/devnet/font/pdfs/5013.Cyrillic_Font_Spec.pdf - http://www.adobe.com/content/dam/Adobe/en/devnet/font/pdfs/5013.Cyrillic_Font_Spec.pdf
too.

The solution works with the two examples I got from Ingo. Maybe there is more to do. But it is a first entry.

Cheers and merry christmas

Werner

Posted By: Ingo
Date Posted: 23 Dec 11 at 12:53PM

Hi Werner!

Thanks a lot for this!
Now i can go further ...
So there's the question which are the most used worldwide encode-formats ;-)

To you and all the other ones here

a Merry Christmas and a Happy New Year,

Ingo

Posted By: edvoigt
Date Posted: 23 Dec 11 at 1:41PM

Hi,

it seems to be a good source here:

http://www.science.co.il/language/locale-codes.asp?s=codepage - http://www.science.co.il/language/locale-codes.asp?s=codepage

There are some other forwarding links.

And you need the relation between afii-codes and copepage.

May it helps.

Werner

Posted By: edvoigt
Date Posted: 29 Dec 11 at 7:53PM

Hi,

here a short adwise how make it not so dirty and more sure.

The solution above determes if a font is using afii-codes. Consequently you have to figure out, in which part of the PDF-Text is which font used. The test only if a such font exists is rather unexact. It may happen, there is a mixture of languages (and codepages) on one page. This is rather sure if there are parts from different PDFs combined to a new one. Therefore there is a journey first through the fontdefinitions and than through the content needed.

Werner