dl::WordPieceTokenizer Class Reference

Implements WordPiece tokenizaton as proposed in [9] using the optimized algorithm by [7] . More...

#include <wordpiece.hpp>

Classes
struct	Conf

Public Types
using	PieceIter = utils::GenericIterator< std::tuple< std::string, std::size_t > >

Public Member Functions
std::vector< size_t >	tokenize (const std::string &text) const noexcept

Static Public Member Functions
static WordPieceTokenizer	fromConf (std::istream &stream) noexcept

Detailed Description

Implements WordPiece tokenizaton as proposed in [9] using the optimized algorithm by [7] .

Definition at line 28 of file wordpiece.hpp.

Member Typedef Documentation

PieceIter

using dl::WordPieceTokenizer::PieceIter = utils::GenericIterator<std::tuple<std::string, std::size_t> >

Definition at line 30 of file wordpiece.hpp.

---

The documentation for this class was generated from the following file:

• dl/model/transformer/wordpiece.hpp