File: //lib/python3/dist-packages/html5lib/__pycache__/_tokenizer.cpython-310.pyc
o
V=�^�, � @ s� d dl mZmZmZ d dlmZ d dlmZm Z d dl
m Z ddl m
Z
ddl mZ ddl mZmZ dd l mZmZmZ dd
l mZmZ dd l mZ dd lmZ dd
lmZ ee�Ze dkrae Z ne Z G dd� de �Z dS )� )�absolute_import�division�unicode_literals)�unichr)�deque� OrderedDict)� version_info� )�spaceCharacters)�entities)� asciiLetters�asciiUpper2Lower)�digits� hexDigits�EOF)�
tokenTypes�
tagTokenTypes)�replacementCharacters)�HTMLInputStream)�Trie)� � c sd e Zd ZdZd�� fdd� Zdd� Zdd� Zd�d
d �Zd d
� Zdd� Z dd� Z
dd� Z dd� Z dd� Z
dd� Zdd� Zd d � Zd d � Zd d!� Zd"d#� Zd$d%� Zd&d'� Zd(d)� Zd*d+� Zd,d-� Zd.d/� Zd0d1� Zd2d3� Zd4d5� Z d6d7� Z d8d9� Z d:d;� Z d<d=� Z d>d?� Z!d@dA� Z"dBdC� Z#dDdE� Z$dFdG� Z%dHdI� Z&dJdK� Z'dLdM� Z(dNdO� Z)dPdQ� Z*dRdS� Z+dTdU� Z,dVdW� Z-dXdY� Z.dZd[� Z/d\d]� Z0d^d_� Z1d`da� Z2dbdc� Z3ddde� Z4dfdg� Z5dhdi� Z6djdk� Z7dldm� Z8dndo� Z9dpdq� Z:drds� Z;dtdu� Z<dvdw� Z=dxdy� Z>dzd{� Z?d|d}� Z@d~d� ZAd�d�� ZBd�d�� ZCd�d�� ZDd�d�� ZEd�d�� ZFd�d�� ZGd�d�� ZHd�d�� ZId�d�� ZJd�d�� ZKd�d�� ZL� ZMS )��
HTMLTokenizera This class takes care of tokenizing HTML.
* self.currentToken
Holds the token that is currently being processed.
* self.state
Holds a reference to the method to be invoked... XXX
* self.stream
Points to HTMLInputStream object.
Nc sJ t |fi |��| _|| _d| _g | _| j| _d| _d | _t t
| �� � d S �NF) r �stream�parser�
escapeFlag�
lastFourChars� dataState�state�escape� currentToken�superr �__init__)�selfr r �kwargs�� __class__� �5/usr/lib/python3/dist-packages/html5lib/_tokenizer.pyr# ( s zHTMLTokenizer.__init__c c sf � t g �| _| �� r1| jjr td | jj�d�d�V | jjs| jr+| j�� V | js"| �� s
dS dS )z� This is where the magic happens.
We do our usually processing through the states and when we have a token
to return we yield the token which pauses processing until the next token
is requested.
�
ParseErrorr ��type�dataN)r �
tokenQueuer r �errorsr �pop�popleft�r$ r( r( r) �__iter__7 s �
� ��zHTMLTokenizer.__iter__c C s� t }d}|r
t}d}g }| j�� }||v r+|tur+|�|� | j�� }||v r+|tustd�|�|�}|tv rJt| }| j �t
d dd|id�� n�d| krTd ksYn |d
krjd }| j �t
d dd|id�� nfd | krtd
ks�n d| krdks�n d| kr�dks�n d| kr�dks�n |t g d��v r�| j �t
d dd|id�� zt |�}W n t
y� |d }t d|d? B �t d|d@ B � }Y nw |dkr�| j �t
d dd�� | j�|� |S )z�This function returns either U+FFFD or the character based on the
decimal or hexadecimal representation. It also discards ";" if present.
If not present self.tokenQueue.append({"type": tokenTypes["ParseError"]}) is invoked.
�
� � r* z$illegal-codepoint-for-numeric-entity� charAsInt�r, r- �datavarsi � i�� � � �r � � � � � i� i� )#� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i��
i��
i�� i�� i�� i�� i��
i��
i�� i�� i�� i�� i�� r: i i � i� �;z numeric-entity-without-semicolonr+ )r r r �charr �append�int�joinr r. r � frozenset�chr�
ValueError�unget) r$ �isHex�allowed�radix� charStack�cr7 rC �vr( r( r) �consumeNumberEntityG s\
�
�
� � $� � z!HTMLTokenizer.consumeNumberEntityFc C s� d}| j �� g}|d tv s!|d tddfv s!|d ur+||d kr+| j �|d � �n|d dkr�d}|�| j �� � |d dv rKd}|�| j �� � |rS|d tv s[|si|d tv ri| j �|d � | �|�}n�| j �t
d d
d �� | j �|� � � dd � |� }n�|d tur�t
�d � |��s�n|�| j �� � |d tus�zt
�d � |d d� ��}t|�}W n ty� d }Y nw |d u�r|d d
kr�| j �t
d dd �� |d d
kr�|r�|| tv s�|| tv s�|| dkr�| j �|� � � dd � |� }n2t| }| j �|� � � |d � ||d � �7 }n| j �t
d dd �� | j �|� � � dd � |� }|�rC| jd d d |7 < d S |tv �rKd}nd}| j �t
| |d �� d S )N�&r �<�#F���)�x�XTr* zexpected-numeric-entityr+ r6 rB z named-entity-without-semicolon�=zexpected-named-entityr- r �SpaceCharacters�
Characters)r rC r
r rJ rD r r rQ r. r r0 rF � entitiesTrie�has_keys_with_prefix�longest_prefix�len�KeyErrorr r r! ) r$ � allowedChar�
fromAttribute�outputrN �hex�
entityName� entityLength� tokenTyper( r( r) �
consumeEntity� st � � � �
� �
zHTMLTokenizer.consumeEntityc C s | j |dd� dS )zIThis method replaces the need for "entityInAttributeValueState".
T)r` ra N)rg )r$ r` r( r( r) �processEntityInAttribute� s z&HTMLTokenizer.processEntityInAttributec C s� | j }|d tv r^|d �t�|d< |d td kr8|d }t|�}t|�t|�kr4|�|ddd� � ||d< |d td kr^|d rO| j� td d d
�� |d r^| j� td d d
�� | j� |� | j
| _ dS )
z�This method is a generic handler for emitting the tags. It also sets
the state to "data" because that's what's needed after a token has been
emitted.
r, �name�StartTagr- NrU �EndTagr* zattributes-in-end-tagr+ � selfClosingz self-closing-flag-on-end-tag) r! r � translater
r � attributeMapr^ �updater. rD r r )r$ �token�rawr- r( r( r) �emitCurrentToken� s( � � z HTMLTokenizer.emitCurrentTokenc C s� | j �� }|dkr| j| _d
S |dkr| j| _d
S |dkr5| j�td dd�� | j�td dd�� d
S |tu r;dS |t v rS| j�td || j �
t d
� d�� d
S | j �
d �}| j�td || d�� d
S ) NrR rS � r* �invalid-codepointr+ rZ FrY T�rR rS rs ) r rC �entityDataStater � tagOpenStater. rD r r r
�
charsUntil�r$ r- �charsr( r( r) r s6
�� � �� � � �zHTMLTokenizer.dataStatec C � | � � | j| _dS �NT)rg r r r2 r( r( r) rv � z HTMLTokenizer.entityDataStatec C s� | j �� }|dkr| j| _d S |dkr| j| _d S |tkr dS |dkr;| j�td dd�� | j�td d d�� d S |t v rS| j�td
|| j �
t d � d�� d S | j �
d �}| j�td || d�� d S )
NrR rS Frs r* rt r+ rZ r; rY Tru ) r rC �characterReferenceInRcdatar �rcdataLessThanSignStater r. rD r r
rx ry r( r( r) � rcdataState" s6
�� � �� � � �zHTMLTokenizer.rcdataStatec C r{ r| )rg r� r r2 r( r( r) r~ ? r} z(HTMLTokenizer.characterReferenceInRcdatac C � | j �� }|dkr| j| _d
S |dkr+| j�td dd�� | j�td dd�� d
S |tkr1dS | j �d �}| j�td || d�� d
S � NrS rs r* rt r+ rZ r; F)rS rs T) r rC �rawtextLessThanSignStater r. rD r r rx ry r( r( r) � rawtextStateD �&
� � � � �zHTMLTokenizer.rawtextStatec C r� r� ) r rC �scriptDataLessThanSignStater r. rD r r rx ry r( r( r) �scriptDataStateV r� z HTMLTokenizer.scriptDataStatec C st | j �� }|tkr dS |dkr'| j�td dd�� | j�td dd�� dS | j�td || j �d� d�� dS ) NFrs r* rt r+ rZ r; T)r rC r r. rD r rx �r$ r- r( r( r) �plaintextStateh s
� � ��z HTMLTokenizer.plaintextStatec C s | j �� }|dkr| j| _dS |dkr| j| _dS |tv r.td |g ddd�| _| j| _dS |dkrN| j �
td dd �� | j �
td
d d �� | j | _dS |d kri| j �
td d
d �� | j � |� | j
| _dS | j �
td dd �� | j �
td
dd �� | j � |� | j | _dS )N�!�/rj F)r, ri r- rl �selfClosingAcknowledged�>r* z'expected-tag-name-but-got-right-bracketr+ rZ z<>�?z'expected-tag-name-but-got-question-markzexpected-tag-namerS T)r rC �markupDeclarationOpenStater �closeTagOpenStater r r! � tagNameStater. rD r rJ �bogusCommentStater� r( r( r) rw w sH
� ��� �� � �� zHTMLTokenizer.tagOpenStatec C s� | j �� }|tv rtd |g dd�| _| j| _dS |dkr.| j�td dd�� | j | _dS |t
u rN| j�td dd�� | j�td d
d�� | j | _dS | j�td d d |id
�� | j � |� | j | _dS )Nrk F�r, ri r- rl r� r* z*expected-closing-tag-but-got-right-bracketr+ z expected-closing-tag-but-got-eofrZ �</z!expected-closing-tag-but-got-charr- r8 T)
r rC r r r! r� r r. rD r r rJ r� r� r( r( r) r� � s6
�� �
� � �� z HTMLTokenizer.closeTagOpenStatec C s� | j �� }|tv r| j| _d
S |dkr| �� d
S |tu r.| j�t d dd�� | j
| _d
S |dkr8| j | _d
S |dkrR| j�t d dd�� | j d d 7 < d
S | j d |7 < d
S ) Nr� r* zeof-in-tag-namer+ r� rs rt ri r; T)
r rC r
�beforeAttributeNameStater rr r r. rD r r �selfClosingStartTagStater! r� r( r( r) r� � s0
�� � � � ��zHTMLTokenizer.tagNameStatec C �R | j �� }|dkrd| _| j| _dS | j�td dd�� | j �|� | j | _dS �Nr� r6 rZ rS r+ T)
r rC �temporaryBuffer�rcdataEndTagOpenStater r. rD r rJ r� r� r( r( r) r � �
� z%HTMLTokenizer.rcdataLessThanSignStatec C �Z | j �� }|tv r| j|7 _| j| _dS | j�td dd�� | j � |� | j
| _dS �NrZ r� r+ T) r rC r r� �rcdataEndTagNameStater r. rD r rJ r� r� r( r( r) r� � �
� z#HTMLTokenizer.rcdataEndTagOpenStatec C � | j o| j d �� | j�� k}| j�� }|tv r+|r+td | jg dd�| _ | j| _d
S |dkrB|rBtd | jg dd�| _ | j | _d
S |dkr]|r]td | jg dd�| _ | �
� | j | _d
S |t v rj| j|7 _d
S | j
�td d| j d �� | j�|� | j| _d
S � Nri rk Fr� r� r� rZ r� r+ T)r! �lowerr� r rC r
r r� r r� rr r r r. rD rJ r� �r$ � appropriater- r( r( r) r� � �B
� �� ��� �� z#HTMLTokenizer.rcdataEndTagNameStatec C r� r� )
r rC r� �rawtextEndTagOpenStater r. rD r rJ r� r� r( r( r) r� � r� z&HTMLTokenizer.rawtextLessThanSignStatec C r� r� ) r rC r r� �rawtextEndTagNameStater r. rD r rJ r� r� r( r( r) r� r� z$HTMLTokenizer.rawtextEndTagOpenStatec C r� r� )r! r� r� r rC r
r r� r r� rr r r r. rD rJ r� r� r( r( r) r� r� z$HTMLTokenizer.rawtextEndTagNameStatec C s| | j �� }|dkrd| _| j| _dS |dkr'| j�td dd�� | j| _dS | j�td dd�� | j � |� | j
| _dS ) Nr� r6 r� rZ z<!r+ rS T) r rC r� �scriptDataEndTagOpenStater r. rD r �scriptDataEscapeStartStaterJ r� r� r( r( r) r� , s
�� z)HTMLTokenizer.scriptDataLessThanSignStatec C r� r� ) r rC r r� �scriptDataEndTagNameStater r. rD r rJ r� r� r( r( r) r� : r� z'HTMLTokenizer.scriptDataEndTagOpenStatec C r� r� )r! r� r� r rC r
r r� r r� rr r r r. rD rJ r� r� r( r( r) r� E r� z'HTMLTokenizer.scriptDataEndTagNameStatec C �L | j �� }|dkr| j�td dd�� | j| _dS | j �|� | j| _dS �N�-rZ r+ T) r rC r. rD r � scriptDataEscapeStartDashStater rJ r� r� r( r( r) r� a �
�z(HTMLTokenizer.scriptDataEscapeStartStatec C r� r� ) r rC r. rD r � scriptDataEscapedDashDashStater rJ r� r� r( r( r) r� k r� z,HTMLTokenizer.scriptDataEscapeStartDashStatec C s� | j �� }|dkr| j�td dd�� | j| _d
S |dkr$| j| _d
S |dkr@| j�td dd�� | j�td dd�� d
S |tkrJ| j | _d
S | j �
d �}| j�td || d�� d
S ) Nr� rZ r+ rS rs r* rt r; )rS r� rs T) r rC r. rD r �scriptDataEscapedDashStater �"scriptDataEscapedLessThanSignStater r rx ry r( r( r) �scriptDataEscapedStateu s0
� � � �� � �z$HTMLTokenizer.scriptDataEscapedStatec C s� | j �� }|dkr| j�td dd�� | j| _d S |dkr$| j| _d S |dkrD| j�td dd�� | j�td dd�� | j| _d S |t krN| j
| _d S | j�td |d�� | j| _d S )
Nr� rZ r+ rS rs r* rt r; T) r rC r. rD r r� r r� r� r r r� r( r( r) r� � s.
� � � ���z(HTMLTokenizer.scriptDataEscapedDashStatec C s� | j �� }|dkr| j�td dd�� d
S |dkr | j| _d
S |dkr5| j�td dd�� | j| _d
S |dkrU| j�td dd�� | j�td d d�� | j| _d
S |t kr_| j
| _d
S | j�td |d�� | j| _d
S ) Nr� rZ r+ rS r� rs r* rt r; T) r rC r. rD r r� r r� r� r r r� r( r( r) r� � s4
�� � � ���z,HTMLTokenizer.scriptDataEscapedDashDashStatec C s� | j �� }|dkrd| _| j| _dS |tv r,| j�td d| d�� || _| j | _dS | j�td dd�� | j �
|� | j | _dS r� ) r rC r� � scriptDataEscapedEndTagOpenStater r r. rD r � scriptDataDoubleEscapeStartStaterJ r� r� r( r( r) r� � s
�� z0HTMLTokenizer.scriptDataEscapedLessThanSignStatec C sR | j �� }|tv r|| _| j| _dS | j�td dd�� | j � |� | j
| _dS r� ) r rC r r� � scriptDataEscapedEndTagNameStater r. rD r rJ r� r� r( r( r) r� � r� z.HTMLTokenizer.scriptDataEscapedEndTagOpenStatec C r� r� )r! r� r� r rC r
r r� r r� rr r r r. rD rJ r� r� r( r( r) r� � r� z.HTMLTokenizer.scriptDataEscapedEndTagNameStatec C � | j �� }|ttd�B v r+| j�td |d�� | j�� dkr%| j | _
dS | j | _
dS |t v rC| j�td |d�� | j|7 _dS | j �
|� | j | _
dS �N)r� r� rZ r+ �scriptT)r rC r
rG r. rD r r� r� � scriptDataDoubleEscapedStater r� r rJ r� r( r( r) r� � �
�� �z.HTMLTokenizer.scriptDataDoubleEscapeStartStatec C s� | j �� }|dkr| j�td dd�� | j| _d
S |dkr/| j�td dd�� | j| _d
S |dkrK| j�td dd�� | j�td dd�� d
S |tkr`| j�td d d�� | j | _d
S | j�td |d�� d
S � Nr� rZ r+ rS rs r* rt r; �eof-in-script-in-scriptT)
r rC r. rD r � scriptDataDoubleEscapedDashStater �(scriptDataDoubleEscapedLessThanSignStater r r� r( r( r) r� � s2
� � � �� ��z*HTMLTokenizer.scriptDataDoubleEscapedStatec C s� | j �� }|dkr| j�td dd�� | j| _d
S |dkr/| j�td dd�� | j| _d
S |dkrO| j�td dd�� | j�td dd�� | j| _d
S |t krd| j�td d d�� | j
| _d
S | j�td |d�� | j| _d
S r� ) r rC r. rD r �$scriptDataDoubleEscapedDashDashStater r� r� r r r� r( r( r) r� s6
�� � �� ��z.HTMLTokenizer.scriptDataDoubleEscapedDashStatec C s | j �� }|dkr| j�td dd�� d S |dkr+| j�td dd�� | j| _d S |dkr@| j�td dd�� | j| _d S |dkr`| j�td dd�� | j�td d d�� | j| _d S |t kru| j�td d
d�� | j
| _d S | j�td |d�� | j| _d S ) Nr� rZ r+ rS r� rs r* rt r; r� T) r rC r. rD r r� r r� r� r r r� r( r( r) r� % s<
��� � �� ��z2HTMLTokenizer.scriptDataDoubleEscapedDashDashStatec C sR | j �� }|dkr | j�td dd�� d| _| j| _dS | j �|� | j | _dS )Nr� rZ r+ r6 T)
r rC r. rD r r� � scriptDataDoubleEscapeEndStater rJ r� r� r( r( r) r� > s
�z6HTMLTokenizer.scriptDataDoubleEscapedLessThanSignStatec C r� r� )r rC r
rG r. rD r r� r� r� r r� r rJ r� r( r( r) r� I r� z,HTMLTokenizer.scriptDataDoubleEscapeEndStatec C s: | j �� }|tv r| j �td� dS |tv r&| jd �|dg� | j| _dS |dkr0| � � dS |dkr:| j
| _dS |dv rY| j �t d dd �� | jd �|dg� | j| _dS |d
krx| j �t d d d �� | jd �d dg� | j| _dS |t
u r�| j �t d d
d �� | j| _dS | jd �|dg� | j| _dS )NTr- r6 r� r� )�'�"rX rS r* �#invalid-character-in-attribute-namer+ rs rt r; z#expected-attribute-name-but-got-eof)r rC r
rx r r! rD �attributeNameStater rr r� r. r r r r� r( r( r) r� Y sJ
���� �
� �� ��z&HTMLTokenizer.beforeAttributeNameStatec C s� | j �� }d}d}|dkr| j| _n�|tv r-| jd d d || j �td� 7 < d}nz|dkr4d}ns|tv r=| j| _nj|dkrF| j | _na|d kre| j
� t d
d d �� | jd d d d
7 < d}nB|dv r�| j
� t d
dd �� | jd d d |7 < d}n#|t
u r�| j
� t d
dd �� | j| _n| jd d d |7 < d}|r�| jd d d �t�| jd d d< | jd d d� D ] \}}| jd d d |kr�| j
� t d
dd �� nq�|r�| �� dS )NTFrX r- rU r r� r� rs r* rt r+ r; �r� r� rS r� zeof-in-attribute-namezduplicate-attribute)r rC �beforeAttributeValueStater r r! rx r
�afterAttributeNameStater� r. rD r r r rm r
rr )r$ r- �leavingThisState� emitTokenri �_r( r( r) r� w s`
�
� � �
� ��z HTMLTokenizer.attributeNameStatec C sN | j �� }|tv r| j �td� dS |dkr | j| _dS |dkr&| �� dS |tv r:| jd � |dg� | j
| _dS |dkrD| j | _dS |dkrc| j � t
d d d
�� | jd � d dg� | j
| _dS |d v r�| j � t
d d
d
�� | jd � |dg� | j
| _dS |tu r�| j � t
d dd
�� | j| _dS | jd � |dg� | j
| _dS )NTrX r� r- r6 r� rs r* rt r+ r; r� z&invalid-character-after-attribute-namez expected-end-of-tag-but-got-eof)r rC r
rx r� r rr r r! rD r� r� r. r r r r� r( r( r) r� � sP
����� �
� �� ��z%HTMLTokenizer.afterAttributeNameStatec C sn | j �� }|tv r| j �td� dS |dkr | j| _dS |dkr,| j| _| j �|� dS |dkr6| j| _dS |dkrK| j �
t d dd�� | � � dS |d krm| j �
t d d
d�� | j
d d d
d7 < | j| _dS |dv r�| j �
t d dd�� | j
d d d
|7 < | j| _dS |tu r�| j �
t d dd�� | j| _dS | j
d d d
|7 < | j| _dS )NTr� rR r� r� r* z.expected-attribute-value-but-got-right-bracketr+ rs rt r- rU r r; )rX rS �`z"equals-in-unquoted-attribute-valuez$expected-attribute-value-but-got-eof)r rC r
rx � attributeValueDoubleQuotedStater �attributeValueUnQuotedStaterJ � attributeValueSingleQuotedStater. rD r rr r! r r r� r( r( r) r� � sV
�� �� �� �
� �� ��z'HTMLTokenizer.beforeAttributeValueStatec C �� | j �� }|dkr| j| _d
S |dkr| �d� d
S |dkr8| j�td dd�� | jd d d d
7 < d
S |t u rM| j�td d d�� | j
| _d
S | jd d d || j � d � 7 < d
S )Nr� rR rs r* rt r+ r- rU r r; z#eof-in-attribute-value-double-quote)r� rR rs T� r rC �afterAttributeValueStater rh r. rD r r! r r rx r� r( r( r) r� � �.
�
� �� ��
�z-HTMLTokenizer.attributeValueDoubleQuotedStatec C r� )Nr� rR rs r* rt r+ r- rU r r; z#eof-in-attribute-value-single-quote)r� rR rs Tr� r� r( r( r) r� r� z-HTMLTokenizer.attributeValueSingleQuotedStatec C s | j �� }|tv r| j| _dS |dkr| �d� dS |dkr$| �� dS |dv rB| j�t d dd�� | j
d d d |7 < dS |d
kr`| j�t d d d�� | j
d d d d 7 < dS |t u ru| j�t d d
d�� | j | _dS | j
d d d || j �
td�tB � 7 < dS )NrR r� )r� r� rX rS r� r* z0unexpected-character-in-unquoted-attribute-valuer+ r- rU r rs rt r; z eof-in-attribute-value-no-quotes)rR r� r� r� rX rS r� rs T)r rC r
r� r rh rr r. rD r r! r r rx rG r� r( r( r) r� s@
�
�� � � �� ��
�z)HTMLTokenizer.attributeValueUnQuotedStatec C s� | j �� }|tv r| j| _dS |dkr| �� dS |dkr#| j| _dS |tu r>| j� t
d dd�� | j � |� | j | _dS | j� t
d dd�� | j � |� | j| _dS )Nr� r� r* z$unexpected-EOF-after-attribute-valuer+ z*unexpected-character-after-attribute-valueT)
r rC r
r� r rr r� r r. rD r rJ r r� r( r( r) r� . s.
�
� � � �� z&HTMLTokenizer.afterAttributeValueStatec C s� | j �� }|dkrd| jd< | �� dS |tu r/| j�td dd�� | j �|� | j | _
dS | j�td dd�� | j �|� | j | _
dS )Nr� Trl r* z#unexpected-EOF-after-solidus-in-tagr+ z)unexpected-character-after-solidus-in-tag) r rC r! rr r r. rD r rJ r r r� r� r( r( r) r� B s$
� � �� z&HTMLTokenizer.selfClosingStartTagStatec C sD | j �d�}|�dd�}| j�td |d�� | j �� | j| _dS )Nr� rs r; �Commentr+ T) r rx �replacer. rD r rC r r r� r( r( r) r� T s �
z HTMLTokenizer.bogusCommentStatec C sn | j �� g}|d dkr)|�| j �� � |d dkr(td dd�| _| j| _dS nq|d dv r\d}dD ]}|�| j �� � |d |vrGd } nq3|r[td
dd d dd �| _| j| _dS n>|d d kr�| jd ur�| jj j
r�| jj j
d j | jj j kr�d}d
D ]}|�| j �� � |d |kr�d } nq}|r�| j
| _dS | j�td dd�� |r�| j �|�� � |s�| j| _dS )NrU r� r� r6 r+ T)�d�D))�o�O�rO �C��t�T��y�Y��p�P��e�EF�Doctype)r, ri �publicId�systemId�correct�[)r� r� �Ar� r� r� r* zexpected-dashes-or-doctype)r rC rD r r! �commentStartStater � doctypeStater �tree� openElements� namespace�defaultNamespace�cdataSectionStater. rJ r0 r� )r$ rN �matched�expectedr( r( r) r� c s` � ���
� � ��z(HTMLTokenizer.markupDeclarationOpenStatec C s� | j �� }|dkr| j| _d S |dkr)| j�td dd�� | jd d7 < d S |dkrE| j�td d d�� | j�| j� | j| _d S |t u ra| j�td d
d�� | j�| j� | j| _d S | jd |7 < | j
| _d S ) Nr� rs r* rt r+ r- r; r� �incorrect-comment�eof-in-commentT) r rC �commentStartDashStater r. rD r r! r r � commentStater� r( r( r) r� � s6
� �� � � ��z HTMLTokenizer.commentStartStatec C s� | j �� }|dkr| j| _d S |dkr)| j�td dd�� | jd d7 < d S |dkrE| j�td d d�� | j�| j� | j| _d S |t u ra| j�td d
d�� | j�| j� | j| _d S | jd d| 7 < | j
| _d S ) Nr� rs r* rt r+ r- � -�r� r� r� T) r rC �commentEndStater r. rD r r! r r r� r� r( r( r) r� � s6
� �� � � ��z#HTMLTokenizer.commentStartDashStatec C s� | j �� }|dkr| j| _d
S |dkr)| j�td dd�� | jd d7 < d
S |tu rE| j�td dd�� | j�| j� | j | _d
S | jd || j �
d � 7 < d
S ) Nr� rs r* rt r+ r- r; r� )r� rs T) r rC �commentEndDashStater r. rD r r! r r rx r� r( r( r) r� � s*
� � � � �
�zHTMLTokenizer.commentStatec C s� | j �� }|dkr| j| _d S |dkr-| j�td dd�� | jd d7 < | j| _d S |t u rI| j�td dd�� | j�| j� | j
| _d S | jd d| 7 < | j| _d S )
Nr� rs r* rt r+ r- r� zeof-in-comment-end-dashT) r rC r� r r. rD r r! r� r r r� r( r( r) r� � s*
� � � ��z!HTMLTokenizer.commentEndDashStatec C s6 | j �� }|dkr| j�| j� | j| _dS |dkr4| j�td dd�� | jd d7 < | j| _dS |dkrI| j�td d d�� | j | _dS |d
krc| j�td d d�� | jd |7 < dS |t
u r| j�td d d�� | j�| j� | j| _dS | j�td d
d�� | jd d| 7 < | j| _dS )Nr� rs r* rt r+ r- u --�r� z,unexpected-bang-after-double-dash-in-commentr� z,unexpected-dash-after-double-dash-in-commentzeof-in-comment-double-dashzunexpected-char-in-commentz--T) r rC r. rD r! r r r r� �commentEndBangStater r� r( r( r) r� � sJ
� �� �� � � � ��z HTMLTokenizer.commentEndStatec C s� | j �� }|dkr| j�| j� | j| _d S |dkr)| jd d7 < | j| _d S |dkrG| j�td dd�� | jd d 7 < | j | _d S |t
u rc| j�td d
d�� | j�| j� | j| _d S | jd d| 7 < | j | _d S ) Nr� r� r- z--!rs r* rt r+ u --!�z eof-in-comment-end-bang-stateT) r rC r. rD r! r r r� r r� r r� r( r( r) r� s4
�� � � ��z!HTMLTokenizer.commentEndBangStatec C s� | j �� }|tv r| j| _dS |tu r0| j�td dd�� d| j d< | j�| j � | j
| _dS | j�td dd�� | j � |� | j| _dS )Nr* �!expected-doctype-name-but-got-eofr+ Fr� zneed-space-after-doctypeT) r rC r
�beforeDoctypeNameStater r r. rD r r! r rJ r� r( r( r) r� s$
� �
�� zHTMLTokenizer.doctypeStatec C s� | j �� }|tv r d S |dkr-| j�td dd�� d| jd< | j�| j� | j| _d S |dkrG| j�td dd�� d | jd
< | j | _d S |t
u rh| j�td d d�� d| jd< | j�| j� | j| _d S || jd
< | j | _d S )
Nr� r* z+expected-doctype-name-but-got-right-bracketr+ Fr� rs rt r; ri r� T) r rC r
r. rD r r! r r �doctypeNameStater r� r( r( r) r * s<
� �
� �
� �
�z$HTMLTokenizer.beforeDoctypeNameStatec C s | j �� }|tv r| jd �t�| jd< | j| _d S |dkr6| jd �t�| jd< | j� | j� | j
| _d S |dkrT| j� t d dd�� | jd d7 < | j | _d S |t
u r�| j� t d dd�� d | jd
< | jd �t�| jd< | j� | j� | j
| _d S | jd |7 < d S ) Nri r� rs r* rt r+ r; zeof-in-doctype-nameFr� T)r rC r
r! rm r
�afterDoctypeNameStater r. rD r r r r r� r( r( r) r D s8
�� �
� �
�z HTMLTokenizer.doctypeNameStatec C sH | j �� }|tv r dS |dkr | j�| j� | j| _dS |tu rDd| jd< | j � |� | j�t
d dd�� | j�| j� | j| _dS |dv red}d D ]}| j �� }||vr[d} nqL|rd| j | _dS n |d
v r�d}d D ]}| j �� }||vr|d} nqm|r�| j | _dS | j � |� | j�t
d d d
|id�� d| jd< | j
| _dS )Nr� Fr� r* �eof-in-doctyper+ r� T))�u�U)�b�B)�l�L)�i�Ir� ��s�S)r� r r� r� )�m�Mz*expected-space-or-right-bracket-in-doctyper- r8 )r rC r
r. rD r! r r r rJ r � afterDoctypePublicKeywordState� afterDoctypeSystemKeywordState�bogusDoctypeState)r$ r- r� r� r( r( r) r ] s`
.�+�
�$�
��
� �
z#HTMLTokenizer.afterDoctypeNameStatec C � | j �� }|tv r| j| _dS |dv r*| j�td dd�� | j �|� | j| _dS |t u rK| j�td dd�� d| j
d< | j�| j
� | j | _dS | j �|� | j| _dS � N)r� r� r* �unexpected-char-in-doctyper+ r Fr� T) r rC r
�"beforeDoctypePublicIdentifierStater r. rD r rJ r r! r r� r( r( r) r � �,
� �
� �
�z,HTMLTokenizer.afterDoctypePublicKeywordStatec C s | j �� }|tv r d
S |dkrd| jd< | j| _d
S |dkr*d| jd< | j| _d
S |dkrK| j�t d dd�� d | jd
< | j�| j� | j
| _d
S |t u rl| j�t d d d�� d | jd
< | j�| j� | j
| _d
S | j�t d d d�� d | jd
< | j | _d
S )Nr� r6 r� r� r� r* �unexpected-end-of-doctyper+ Fr� r r T)
r rC r
r! �(doctypePublicIdentifierDoubleQuotedStater �(doctypePublicIdentifierSingleQuotedStater. rD r r r r r� r( r( r) r � �D
�
�
� �
� �
��
z0HTMLTokenizer.beforeDoctypePublicIdentifierStatec C �� | j �� }|dkr| j| _d
S |dkr)| j�td dd�� | jd d7 < d
S |dkrJ| j�td d d�� d
| jd < | j�| j� | j| _d
S |t u rk| j�td d d�� d
| jd < | j�| j� | j| _d
S | jd |7 < d
S )Nr� rs r* rt r+ r� r; r� r Fr� r T�
r rC �!afterDoctypePublicIdentifierStater r. rD r r! r r r� r( r( r) r � �8
� �� �
� �
�z6HTMLTokenizer.doctypePublicIdentifierDoubleQuotedStatec C r )Nr� rs r* rt r+ r� r; r� r Fr� r Tr r� r( r( r) r � r z6HTMLTokenizer.doctypePublicIdentifierSingleQuotedStatec C s | j �� }|tv r| j| _d S |dkr | j�| j� | j| _d S |dkr:| j�t d dd�� d| jd< | j
| _d S |dkrT| j�t d dd�� d| jd< | j | _d S |t u ru| j�t d d d�� d
| jd < | j�| j� | j| _d S | j�t d dd�� d
| jd < | j
| _d S )
Nr� r� r* r r+ r6 r� r� r Fr� T)r rC r
�-betweenDoctypePublicAndSystemIdentifiersStater r. rD r! r r �(doctypeSystemIdentifierDoubleQuotedState�(doctypeSystemIdentifierSingleQuotedStater r r� r( r( r) r � sH
�� �
� �
� �
��
z/HTMLTokenizer.afterDoctypePublicIdentifierStatec C s� | j �� }|tv r d S |dkr | j�| j� | j| _d S |dkr,d| jd< | j| _d S |dkr;d| jd< | j | _d S |t
kr\| j�t d dd�� d | jd
< | j�| j� | j| _d S | j�t d d d�� d | jd
< | j | _d S )
Nr� r� r6 r� r� r* r r+ Fr� r T)
r rC r
r. rD r! r r r" r# r r r r� r( r( r) r! s<
��
�
� �
��
z;HTMLTokenizer.betweenDoctypePublicAndSystemIdentifiersStatec C r r ) r rC r
�"beforeDoctypeSystemIdentifierStater r. rD r rJ r r! r r� r( r( r) r ) r z,HTMLTokenizer.afterDoctypeSystemKeywordStatec C s | j �� }|tv r d S |dkrd| jd< | j| _d S |dkr*d| jd< | j| _d S |dkrK| j�t d dd�� d | jd
< | j�| j� | j
| _d S |t u rl| j�t d d d�� d | jd
< | j�| j� | j
| _d S | j�t d dd�� d | jd
< | j | _d S )
Nr� r6 r� r� r� r* r r+ Fr� r T)
r rC r
r! r" r r# r. rD r r r r r� r( r( r) r$ = r z0HTMLTokenizer.beforeDoctypeSystemIdentifierStatec C r )Nr� rs r* rt r+ r� r; r� r Fr� r T�
r rC �!afterDoctypeSystemIdentifierStater r. rD r r! r r r� r( r( r) r" Z r z6HTMLTokenizer.doctypeSystemIdentifierDoubleQuotedStatec C r )Nr� rs r* rt r+ r� r; r� r Fr� r Tr% r� r( r( r) r# r r z6HTMLTokenizer.doctypeSystemIdentifierSingleQuotedStatec C s� | j �� }|tv r dS |dkr | j�| j� | j| _dS |tu r>| j�t d dd�� d| jd< | j�| j� | j| _dS | j�t d dd�� | j
| _dS ) Nr� r* r r+ Fr� r T) r rC r
r. rD r! r r r r r r� r( r( r) r& � s*
� � �
��z/HTMLTokenizer.afterDoctypeSystemIdentifierStatec C s` | j �� }|dkr| j�| j� | j| _dS |tu r-| j �|� | j�| j� | j| _dS dS )Nr� T) r rC r. rD r! r r r rJ r� r( r( r) r � s
� �z HTMLTokenizer.bogusDoctypeStatec C s� g } |� | j�d�� |� | j�d�� | j�� }|tkr n!|dks%J �|d dd � dkr:|d d d� |d< n|� |� qd�|�}|�d�}|d krft|�D ]
}| j� t d
d d �� qR|�
dd
�}|rs| j� t d |d �� | j | _ dS )NT�]r� rU ���z]]r6 rs r r* rt r+ r; rZ )
rD r rx rC r rF �count�ranger. r r� r r )r$ r- rC � nullCountr� r( r( r) r� � s6
�
� �z HTMLTokenizer.cdataSectionState)Nr )N�__name__�
__module__� __qualname__�__doc__r# r3 rQ rg rh rr r rv r� r~ r� r� r� rw r� r� r r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r r r r r r r r r! r r$ r" r# r& r r� �
__classcell__r( r( r& r) r s�
HP #
6 "- 3 r N) �
__future__r r r �sixr rH � collectionsr r �sysr � constantsr
r r r
r r r r r r � _inputstreamr �_trier r[ �dictrn �objectr r( r( r( r) �<module> s"