sharktopus

Sharktopus is an open-source Python library designed to crop GRIB2 weather data in the cloud before download, addressing the challenge of handling large meteorological datasets by reducing their size and transfer time. It enables users to extract subsets of GRIB2 files based on bounding box, variables, and vertical levels, so only the necessary data is retrieved. 25°) end-to-end, with plans for additional open-data products such as HRRR, NAM, RAP, and ECMWF as the community grows. Sharktopus is product-agnostic in its architecture, making it straightforward to add new data sources by plugging in a URL resolver and catalog rather than modifying the core system.

The platform provides three main usage modes: a command-line interface suitable for shell pipelines and automation, a Python API for integration into scripts and data pipelines (including xarray support and batch-level parallelism controls), and an optional local web UI for users who prefer a graphical interface. The web UI allows job submission, quota monitoring, credential management, and inventory browsing, and is designed to run safely on a local machine by binding only to localhost. Sharktopus deploys a serverless wgrib2 worker to cloud platforms such as AWS Lambda, Google Cloud Run, or Azure Container Apps, with each user operating on their own cloud account and covering their own typically minimal cloud costs.

Typical use cases include preparing input data for regional atmospheric models like WRF and MPAS-regional, oceanographic and coupled models such as ROMS, HYCOM, FVCOM, SWAN/WW3, and ADCIRC, as well as machine learning and nowcasting pipelines for applications like precipitation, fire, or air-quality modeling. The tool also supports analysis and teaching scenarios where smaller regional subsets are needed across many cycles without the overhead of downloading full datasets. By cropping data in the cloud, users can reduce a typical 12 GB transfer to around 200 MB and cut wall time from about 20 minutes to 30 seconds for a 72-hour regional domain.

Sharktopus is distributed under the MIT License and is maintained as an independent open-source project. Governance is merit-based and contributors retain their institutional affiliations. The tool was originally developed to support the CONVECT project, coordinated by Dr. Tânia Ocimoto Oda, but it is not a product of or endorsed by any specific institution.