File: //usr/libexec/oracle-cloud-agent/plugins/osms/charset_normalizer/utils.pyc
a
ٓ�fv. � @ sl d dl Z d dlZd dlZd dlmZ d dlmZ d dlmZ d dl m
Z
d dl m Z m
Z
mZmZmZmZ d dlmZ dd lmZmZmZmZmZmZ eed
�ee d �d d
��Z eed
�eed �dd��Z eed
�eee d �dd��Z eed
�ee d �dd��Z eed
�ee d �dd��Z!eed
�ee d �dd��Z"eed
�ee d �dd��Z#eed
�ee d �dd��Z$eed
�ee d �d d ��Z%eed
�ee d �d d ��Z&eed
�ee d �d d!��Z'eed
�ee d �d"d#��Z(eed
�ee d �d$d%��Z)eed
�ee d �d&d'��Z*eed
�ee d �d(d)��Z+eed
�ee d �d*d+��Z,ee-e�d
�ee d,�d-d.��Z.eed
�ee d �d/d0��Z/dSe0e1ee d2�d3d4�Z2ed5d
�ee d6�d7d8��Z3e0eee e0f d9�d:d;�Z4ee d<�d=d>�Z5dTee ed@�dAdB�Z6ee
e dC�dDdE�Z7eee8dF�dGdH�Z9eee dF�dIdJ�Z:dKej;dLfee1eddM�dNdO�Z<dUe0ee=e1e e e0e ee e eddf dP�
dQdR�Z>dS )V� N)�IncrementalDecoder)�aliases)� lru_cache)�findall)� Generator�List�Optional�Set�Tuple�Union)�MultibyteIncrementalDecoder� )�ENCODING_MARKS�IANA_SUPPORTED_SIMILAR� RE_POSSIBLE_ENCODING_INDICATION�UNICODE_RANGES_COMBINED� UNICODE_SECONDARY_RANGE_KEYWORD�UTF8_MAXIMAL_ALLOCATION)�maxsize)� character�returnc C sd zt �| �}W n ty" Y dS 0 d|v pbd|v pbd|v pbd|v pbd|v pbd|v pbd|v pbd |v S )
NFz
WITH GRAVEz
WITH ACUTEz WITH CEDILLAzWITH DIAERESISzWITH CIRCUMFLEXz
WITH TILDEz WITH MACRONzWITH RING ABOVE�� unicodedata�name�
ValueError�r � description� r �|/sparta/input/_build_configuration/image_build+validate/lib/bmcenv/lib64/python3.9/site-packages/charset_normalizer/utils.py�is_accentuated s& �������r c C s. t �| �}|s| S |�d�}tt|d d��S )N� r � )r �
decomposition�split�chr�int)r �
decomposed�codesr r r �
remove_accent( s
r( c C s. t | �}t�� D ]\}}||v r| S qdS )zK
Retrieve the Unicode range official name from a single character.
N)�ordr �items)r �
character_ord�
range_name� ord_ranger r r �
unicode_range3 s
r. c C s, zt �| �}W n ty" Y dS 0 d|v S )NF�LATINr r r r r �is_latinA s
r0 c C s2 t �| �}d|v rdS t| �}|d u r*dS d|v S )N�PTF� Punctuation�r �categoryr. �r �character_category�character_ranger r r �is_punctuationJ s
r8 c C sB t �| �}d|v sd|v r dS t| �}|d u r2dS d|v o@|dkS )N�S�NTF�Forms�Lor3 r5 r r r � is_symbolY s
r= c C s$ t | �}|d u rdS d|v p"d|v S )NF� Emoticons� Pictographs)r. )r r7 r r r � is_emoticonh s r@ c C s. | � � s| dv rdS t�| �}d|v p,|dv S )N> �+u ｜�>�<T�Z> �Pd�Po�Pc)�isspacer r4 )r r6 r r r � is_separatorr s
rI c C s | � � | �� kS �N)�islower�isupper�r r r r �is_case_variable| s rN c C s, zt �| �}W n ty" Y dS 0 d|v S )NF�CJKr �r �character_namer r r �is_cjk� s
rR c C s, zt �| �}W n ty" Y dS 0 d|v S )NF�HIRAGANAr rP r r r � is_hiragana� s
rT c C s, zt �| �}W n ty" Y dS 0 d|v S )NF�KATAKANAr rP r r r � is_katakana� s
rV c C s, zt �| �}W n ty" Y dS 0 d|v S )NF�HANGULr rP r r r � is_hangul� s
rX c C s, zt �| �}W n ty" Y dS 0 d|v S )NF�THAIr rP r r r �is_thai� s
rZ c C s, zt �| �}W n ty" Y dS 0 d|v S )NF�ARABICr rP r r r � is_arabic� s
r\ c C s4 zt �| �}W n ty" Y dS 0 d|v o2d|v S )NFr[ z
ISOLATED FORMr rP r r r �is_arabic_isolated_form� s
r] )r, r c s t � fdd�tD ��S )Nc 3 s | ]}|� v V qd S rJ r )�.0�keyword�r, r r � <genexpr>� � z-is_unicode_range_secondary.<locals>.<genexpr>)�anyr r` r r` r �is_unicode_range_secondary� s rd c C s( | � � du o&| �� du o&| dko&| dkS )NF�u )rH � isprintablerM r r r �is_unprintable� s
���rg � )�sequence� search_zoner c C s� t | t�st�t| �}tt| dt||�� jddd��}t|�dkrHdS |D ]N}|�� � dd�}t
� � D ]0\}}||kr�| S ||krh| S qhqLdS )zW
Extract using ASCII-only decoder any specified encoding in the first n-bytes.
N�ascii�ignore��errorsr �-�_) �
isinstance�bytes� TypeError�lenr r �min�decode�lower�replacer r* )ri rj �seq_len�results�specified_encoding�encoding_alias�
encoding_ianar r r �any_specified_encoding� s"
� r~ � )r r c C s | dv p t t�d�| ��jt�S )zQ
Verify is a specific encoding is a multi byte one based on it IANA name
> � utf_16_le� utf_32_le�utf_32�utf_16�utf_7�utf_8� utf_16_be� utf_32_be� utf_8_sig� encodings.{})�
issubclass� importlib�
import_module�formatr r )r r r r �is_multi_byte_encoding� s
��r� )ri r c C sJ t D ]@}t | }t|t�r |g}|D ] }| �|�r$||f S q$qdS )z9
Identify and extract SIG/BOM in given sequence.
)Nrb )r rq rr �
startswith)ri �
iana_encoding�marks�markr r r �identify_sig_or_bom s
r� )r� r c C s | dvS )N> r� r� r )r� r r r �should_strip_sig_or_bom s r� T)�cp_name�strictr c C sL | � � �dd�} t�� D ] \}}| ||fv r| S q|rHtd�| ���| S )Nro rp z Unable to retrieve IANA for '{}')rw rx r r* r r� )r� r� r| r} r r r � iana_name" s
r� )�decoded_sequencer c C s4 t � }| D ] }t|�}|d u r q
|�|� q
t|�S rJ )�setr. �add�list)r� �rangesr r7 r r r �
range_scan2 s r� )� iana_name_a� iana_name_br c C s� t | �st |�rdS t�d�| ��j}t�d�|��j}|dd�}|dd�}d}td�D ]*}t|g�}|�|�|�|�krX|d7 }qX|d S ) Ng r� rl rm r � r
� )r� r� r� r� r �rangerr rv ) r� r� � decoder_a� decoder_b�id_a�id_b�character_match_count�i�
to_be_decodedr r r �
cp_similarity@ s ��
r� c C s | t v o|t | v S )z�
Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
the function cp_similarity.
)r )r� r� r r r �
is_cp_similarX s
�r� �charset_normalizerz)%(asctime)s | %(levelname)s | %(message)s)r �level�
format_stringr c C s: t �| �}|�|� t �� }|�t �|�� |�|� d S rJ )�logging� getLogger�setLevel�
StreamHandler� setFormatter� Formatter�
addHandler)r r� r� �logger�handlerr r r �set_logging_handlerc s
r� )
� sequencesr} �offsets�
chunk_size�bom_or_sig_available�strip_sig_or_bom� sig_payload�is_multi_byte_decoder�decoded_payloadr c c s* |r6|du r6|D ]"} || | | � }
|
s, q4|
V qn�|D ]�} | | } | t | �d krXq:| | | | � } |r||du r||| } | j||r�dndd�}
|�r | dk�r t|d�}
|�r |
d |
� |v�r t| | d d �D ]H}| || � } |r�|du r�|| } | j|dd�}
|
d |
� |v r� �q q�|
V q:d S )
NF� rl r� rm r r! � ���)rt rv ru r� )r� r} r� r� r� r� r� r� r� r� �chunk� chunk_end� cut_sequence�chunk_partial_size_chk�jr r r �cut_sequence_chunksp s>
�
�� r� )rh )T)N)?r� r� r �codecsr �encodings.aliasesr � functoolsr �rer �typingr r r r r
r �_multibytecodecr �constantr r r r r r �str�boolr r( r. r0 r8 r= r@ rI rN rR rT rV rX rZ r\ r] rt rd rg rr r% r~ r� r� r� r� r� �floatr� r� �INFOr� r� r� r r r r �<module> s�
�� � �