pytorch-tokenizers is an open-source package offering a suite of common tokenizers for use in Python and C++ machine learning workflows, particularly for large language models. It is designed for developers and researchers working with PyTorch and related ML frameworks.
pytorch-tokenizers sits in PulseGate's Frameworks & SDKs category. It focuses on providing efficient and standardized tokenization for machine learning and LLM applications. It is built as an open-source project for machine learning developers and researchers. pytorch-tokenizers is open source under the BSD-3-Clause license. The product ships for the web, the command line, and API.
Behind pytorch-tokenizers is Meta / PyTorch, and the product first shipped in 2024. Development happens publicly on GitHub with 50 stars and 17 commits in the last 90 days. PulseGate's similarity index finds few close equivalents — pytorch-tokenizers occupies a relatively distinct niche. Key capabilities include tokenization, python support, and C++ support.
Latest indexed changes and source events
pytorch.org discovered by the PulseGate indexer
Other apps tracked under the same category.
