Automattic\WooCommerce\Vendor\Symfony\Component\CssSelector\Parser\Tokenizer

Tokenizer::tokenize │ public │ WC 1.0

Tokenize selector source code.

Method of the class: Tokenizer{}

No Hooks.

Returns

null. Nothing (null).

Usage

$Tokenizer = new Tokenizer();
$Tokenizer->tokenize( $reader ): TokenStream;

$reader(Reader) (required)

.

Tokenizer::tokenize() Tokenizer::tokenize code ^{WC 10.5.0}

public function tokenize(Reader $reader): TokenStream
{
    $stream = new TokenStream();

    while (!$reader->isEOF()) {
        foreach ($this->handlers as $handler) {
            if ($handler->handle($reader, $stream)) {
                continue 2;
            }
        }

        $stream->push(new Token(Token::TYPE_DELIMITER, $reader->getSubstring(1), $reader->getPosition()));
        $reader->moveForward(1);
    }

    return $stream
        ->push(new Token(Token::TYPE_FILE_END, null, $reader->getPosition()))
        ->freeze();
}