Petals

Petals is a system for running large language models at home in a decentralized manner. It allows users to generate text and fine-tune models such as Llama 3.1 up to 405 billion parameters, Mixtral 8x22B, Falcon 40B and larger, and BLOOM 176B. The approach follows a BitTorrent-style peer-to-peer network where each participant loads a portion of the model and contributes by serving other parts to the network.

Inference performance reaches up to six tokens per second for Llama 2 70B in single-batch mode and up to four tokens per second for Falcon 180B. This speed supports use in chatbots and interactive applications. The system extends beyond standard API interfaces by permitting any fine-tuning and sampling methods, custom paths through the model, and access to hidden states. It combines the convenience of an API with the flexibility of PyTorch and Hugging Face Transformers.

Petals is offered as a tool that can run on consumer-grade GPUs or within Google Colab notebooks. It forms part of the BigScience research workshop. Documentation and source code are available on GitHub, with development discussion taking place in Discord. A network status display shows current top contributors who are providing GPU resources.