Package encutils :: Module encutils
[hide private]
[frames] | no frames]

Module encutils

source code

Helper functions to detect encodings of text files.

encutils

Website: http://cthedot.de/encutils/

Some basic helper functions to deal with encodings of text files (like HTML, XHTML, XML, CSS, etc.) via HTTP and directly.

getEncodingInfo is probably the main function of interest which uses other supplied functions itself and gathers all information together and supplies an EncodingInfo object with the following properties:

Requires Python 2.3 or later

references

XML

RFC 3023 (http://www.ietf.org/rfc/rfc3023.txt)

easier explained in
HTML
http://www.w3.org/TR/REC-html40/charset.html#h-5.2.2
TODO:
  • HTML meta elements in comments? (use HTMLParser?)
  • parse @charset of HTML elements?
  • check for more texttypes if only text given



Author: Christof Hoeke

License: This work is licensed under a Creative Commons License http://creativecommons.org/licenses/by/2.5/

Classes [hide private]
  EncodingInfo
All encoding related information, returned by getEncodingInfo

Functions [hide private]
  buildlog(logname='encutils', level='INFO', stream=sys.stderr, filename=None, filemode="w", format='%(levelname)s\t%(message)s')
helper to build a basic log
  _getTextTypeByMediaType(media_type, log=None)
returns type as defined by constants above
  _getTextType(text, log=None)
checks if given text is XML (naive test!) used if no content-type given
  encodingByMediaType(media_type, log=None)
Returns a default encoding for the given media_type.
  getHTTPInfo(response, log=None)
Returns (media_type, encoding) information from the response' Content-Type HTTP header.
  getMetaInfo(text, log=None)
Returns (media_type, encoding) information from (first) X/HTML Content-Type <meta> element if available.
  detectXMLEncoding(fp, log=None)
Attempts to detect the character encoding of the xml file given by a file object fp.
  tryEncodings(text, log=None)
If installed uses chardet http://chardet.feedparser.org/ to detect encoding, else tries different encodings on text and returns the one that does not raise an exception which is not very advanced or may be totally wrong.
  getEncodingInfo(response=None, text=u'', log=None)
Finds all encoding related information in given text.

Variables [hide private]
  __version__ = '0.7a1'
  True = True
  False = False
  _XML_APPLICATION_TYPE = 0
application/xml, application/xml-dtd, application/xml-external-parsed-entity, or a subtype like application/rss+xml.
  _XML_TEXT_TYPE = 1
text/xml, text/xml-external-parsed-entity, or a subtype like text/AnythingAtAll+xml
  _HTML_TEXT_TYPE = 2
text/html
  _TEXT_TYPE = 3
any other of text/* like text/plain, text/css, ...
  _OTHER_TYPE = 4
types not fitting in above types

Function Details [hide private]

buildlog(logname='encutils', level='INFO', stream=sys.stderr, filename=None, filemode="w", format='%(levelname)s\t%(message)s')

source code 

helper to build a basic log

  • if filename is given returns a log logging to filename with mode filemode
  • else uses a log streaming to stream which defaults to sys.stderr
  • level defines the level of the log
  • format defines the formatter format of the log

returns a log with the name logname

_getTextTypeByMediaType(media_type, log=None)

source code 
returns type as defined by constants above

_getTextType(text, log=None)

source code 
checks if given text is XML (naive test!) used if no content-type given

encodingByMediaType(media_type, log=None)

source code 

Returns a default encoding for the given media_type. For example 'utf-8' for media_type='application/xml'.

Refers to RFC 3023 and HTTP MIME specification.

If no default encoding is available returns None.

getHTTPInfo(response, log=None)

source code 
Returns (media_type, encoding) information from the response' Content-Type HTTP header. (Case of headers is ignored.) May be (None, None) e.g. if no Content-Type header is available.

getMetaInfo(text, log=None)

source code 

Returns (media_type, encoding) information from (first) X/HTML Content-Type <meta> element if available.

Normally in X/HTML:
<meta http-equiv="Content-Type" content="media_type; charset=encoding"/>

detectXMLEncoding(fp, log=None)

source code 

Attempts to detect the character encoding of the xml file given by a file object fp. fp must not be a codec wrapped file object! fp may also be a string or unicode string

The return value can be:
  • if detection of the BOM succeeds, the codec name of the corresponding unicode charset is returned
  • if BOM detection fails, the xml declaration is searched for the encoding attribute and its value returned. the "<" character has to be the very first in the file then (it's xml standard after all).
  • if BOM and xml declaration fail, utf-8 is returned according to XML 1.0.

Based on a recipe by Lars Tiede: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/363841 which itself is based on Paul Prescotts recipe: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52257

tryEncodings(text, log=None)

source code 

If installed uses chardet http://chardet.feedparser.org/ to detect encoding, else tries different encodings on text and returns the one that does not raise an exception which is not very advanced or may be totally wrong.

Returns working encoding or None if no encoding does work at all.

The returned encoding might nevertheless be not the one intended by the author as it is only checked if the text might be encoded in that encoding. Some texts might be working in "iso-8859-1" and "windows-1252" and "ascii" and "utf-8" and ...

getEncodingInfo(response=None, text=u'', log=None)

source code 

Finds all encoding related information in given text. Uses information in headers of supplied HTTPResponse, possible XML declaration and X/HTML <meta> elements. text will mostly be HTML or XML.

For certain text mismatches may be reported which are not really mismatches. These false warning appear if e.g. a HTTP mime-type of text/html is sent (which is also used for XHTML sometimes) and HTML is actually served. In this case the XML default of 'utf-8' which is not relevant may nevertheless be reported to mismatch with HTTP or <meta>-element information.

Parameters
  • response: HTTP response object, e.g. urllib.urlopen('url')
  • text: to guess encoding for, might include XML prolog with encoding pseudo attribute or HTML meta element
  • log: an optional logging logger to which messages may go, if no log given all log messages are available from resulting EncodingInfo

Returns instance of EncodingInfo.

How the resulting encoding is retrieved

XML

RFC 3023 states if media type given in the Content-Type HTTP header is application/xml, application/xml-dtd, application/xml-external-parsed-entity, or any one of the subtypes of application/xml such as application/atom+xml or application/rss+xml etc then the character encoding is determined in this order:

1. the encoding given in the charset parameter of the Content-Type HTTP header, or 2. the encoding given in the encoding attribute of the XML declaration within the document, or 3. utf-8.

Mismatch possibilities:
  • HTTP + XMLdecla
  • HTTP + HTMLmeta
application/xhtml+xml ?
XMLdecla + HTMLmeta

If the media type given in the Content-Type HTTP header is text/xml, text/xml-external-parsed-entity, or a subtype like text/Anything+xml, the encoding attribute of the XML declaration is ignored completely and the character encoding is determined in the order: 1. the encoding given in the charset parameter of the Content-Type HTTP header, or 2. ascii.

Mismatch possibilities:
  • HTTP + XMLdecla
  • HTTP + HTMLmeta
text/xhtml+xml
XMLdecla + HTMLmeta

HTML

For HTML served as text/html:
http://www.w3.org/TR/REC-html40/charset.html#h-5.2.2
  1. An HTTP "charset" parameter in a "Content-Type" field. (maybe defaults to ISO-8859-1, but should not assume this)
  2. A META declaration with "http-equiv" set to "Content-Type" and a value set for "charset".
  3. The charset attribute set on an element that designates an external resource. (NOT IMPLEMENTED HERE YET)
Mismatch possibilities:
  • HTTP + HTMLmeta

Variables Details [hide private]

__version__

None
Value:
'0.7a1'                                                                
      

True

None
Value:
True                                                                   
      

False

None
Value:
False                                                                  
      

_XML_APPLICATION_TYPE

application/xml, application/xml-dtd, application/xml-external-parsed-entity, or a subtype like application/rss+xml.
Value:
0                                                                     
      

_XML_TEXT_TYPE

text/xml, text/xml-external-parsed-entity, or a subtype like text/AnythingAtAll+xml
Value:
1                                                                     
      

_HTML_TEXT_TYPE

text/html
Value:
2                                                                     
      

_TEXT_TYPE

any other of text/* like text/plain, text/css, ...
Value:
3                                                                     
      

_OTHER_TYPE

types not fitting in above types
Value:
4