Home | Trees | Indices | Help |
|
---|
|
Helper functions to detect encodings of text files.
Website: http://cthedot.de/encutils/
Some basic helper functions to deal with encodings of text files (like HTML, XHTML, XML, CSS, etc.) via HTTP and directly.
getEncodingInfo is probably the main function of interest which uses other supplied functions itself and gathers all information together and supplies an EncodingInfo object with the following properties:
Encoding is the explicit or implicit encoding or None and always lowercase.
Requires Python 2.3 or later
RFC 3023 (http://www.ietf.org/rfc/rfc3023.txt)
Author: Christof Hoeke
License: This work is licensed under a Creative Commons License http://creativecommons.org/licenses/by/2.5/
|
|||
|
EncodingInfo All encoding related information, returned by getEncodingInfo |
|
|||
|
buildlog(logname='encutils',
level='INFO',
stream=sys.stderr,
filename=None,
filemode="w",
format='%(levelname)s\t%(message)s') helper to build a basic log |
||
|
_getTextTypeByMediaType(media_type,
log=None) returns type as defined by constants above |
||
|
_getTextType(text,
log=None) checks if given text is XML (naive test!) used if no content-type given |
||
|
encodingByMediaType(media_type,
log=None) Returns a default encoding for the given media_type. |
||
|
getHTTPInfo(response,
log=None) Returns (media_type, encoding) information from the response' Content-Type HTTP header. |
||
|
getMetaInfo(text,
log=None) Returns (media_type, encoding) information from (first) X/HTML Content-Type <meta> element if available. |
||
|
detectXMLEncoding(fp,
log=None) Attempts to detect the character encoding of the xml file given by a file object fp. |
||
|
tryEncodings(text,
log=None) If installed uses chardet http://chardet.feedparser.org/ to detect encoding, else tries different encodings on text and returns the one that does not raise an exception which is not very advanced or may be totally wrong. |
||
|
getEncodingInfo(response=None,
text=u'',
log=None) Finds all encoding related information in given text. |
|
|||
|
__version__ = '0.7a1'
|
||
|
True = True
|
||
|
False = False
|
||
|
_XML_APPLICATION_TYPE = 0 application/xml, application/xml-dtd, application/xml-external-parsed-entity, or a subtype like application/rss+xml. |
||
|
_XML_TEXT_TYPE = 1 text/xml, text/xml-external-parsed-entity, or a subtype like text/AnythingAtAll+xml |
||
|
_HTML_TEXT_TYPE = 2 text/html |
||
|
_TEXT_TYPE = 3 any other of text/* like text/plain, text/css, ... |
||
|
_OTHER_TYPE = 4 types not fitting in above types |
|
helper to build a basic log
returns a log with the name logname |
|
|
Returns a default encoding for the given media_type. For example 'utf-8' for media_type='application/xml'. Refers to RFC 3023 and HTTP MIME specification. If no default encoding is available returns None. |
|
Returns (media_type, encoding) information from (first) X/HTML Content-Type <meta> element if available.
|
Attempts to detect the character encoding of the xml file given by a file object fp. fp must not be a codec wrapped file object! fp may also be a string or unicode string
Based on a recipe by Lars Tiede: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/363841 which itself is based on Paul Prescotts recipe: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52257 |
If installed uses chardet http://chardet.feedparser.org/ to detect encoding, else tries different encodings on text and returns the one that does not raise an exception which is not very advanced or may be totally wrong. Returns working encoding or None if no encoding does work at all. The returned encoding might nevertheless be not the one intended by the author as it is only checked if the text might be encoded in that encoding. Some texts might be working in "iso-8859-1" and "windows-1252" and "ascii" and "utf-8" and ... |
Finds all encoding related information in given text. Uses information in headers of supplied HTTPResponse, possible XML declaration and X/HTML <meta> elements. text will mostly be HTML or XML. For certain text mismatches may be reported which are not really mismatches. These false warning appear if e.g. a HTTP mime-type of text/html is sent (which is also used for XHTML sometimes) and HTML is actually served. In this case the XML default of 'utf-8' which is not relevant may nevertheless be reported to mismatch with HTTP or <meta>-element information.
Returns instance of EncodingInfo. How the resulting encoding is retrievedXMLRFC 3023 states if media type given in the Content-Type HTTP header is application/xml, application/xml-dtd, application/xml-external-parsed-entity, or any one of the subtypes of application/xml such as application/atom+xml or application/rss+xml etc then the character encoding is determined in this order: 1. the encoding given in the charset parameter of the Content-Type HTTP header, or 2. the encoding given in the encoding attribute of the XML declaration within the document, or 3. utf-8.
If the media type given in the Content-Type HTTP header is text/xml, text/xml-external-parsed-entity, or a subtype like text/Anything+xml, the encoding attribute of the XML declaration is ignored completely and the character encoding is determined in the order: 1. the encoding given in the charset parameter of the Content-Type HTTP header, or 2. ascii.
HTML
|
|
__version__None
|
TrueNone
|
FalseNone
|
_XML_APPLICATION_TYPEapplication/xml, application/xml-dtd, application/xml-external-parsed-entity, or a subtype like application/rss+xml.
|
_XML_TEXT_TYPEtext/xml, text/xml-external-parsed-entity, or a subtype like text/AnythingAtAll+xml
|
_HTML_TEXT_TYPEtext/html
|
_TEXT_TYPEany other of text/* like text/plain, text/css, ...
|
_OTHER_TYPEtypes not fitting in above types
|
Home | Trees | Indices | Help |
|
---|
Generated by Epydoc 3.0alpha3 on Sun Nov 26 19:37:11 2006 | http://epydoc.sourceforge.net |