Segment the HAB

Abstract

Climate change is intensifying the occurrence of harmful algal blooms (HAB), particularly cyanobacteria, which threaten aquatic ecosystems and human health through oxygen depletion, toxin release, and disruption of marine biodiversity. Traditional monitoring approaches, such as manual water sampling, remain labor-intensive and limited in spatial and temporal coverage. Recent advances in vision-language models (VLMs) for remote sensing have shown potential for scalable AI-driven solutions, yet challenges remain in reasoning over imagery and quantifying bloom severity.

In this work, we introduce ALGOS (ALGae Observation and Segmentation), a segmentation-and-reasoning system for HAB monitoring that combines remote sensing image understanding with severity estimation. Our approach integrates GeoSAM-assisted human evaluation for high-quality segmentation mask curation and fine-tunes vision language models on severity prediction using the Cyanobacteria Aggregated Manual Labels (CAML) from NASA. Experiments demonstrate that ALGOS achieves robust performance on both segmentation and severity-level estimation, paving the way toward practical and automated cyanobacterial monitoring systems.

Figure 1: ALGOS pipeline showing multimodal LLM with SAM decoder

Figure 1: The pipeline of ALGOS. Given the input image and text query, the multimodal LLM generates text output. The last-layer embedding for the <SEG> token is then decoded into the segmentation mask via the SAM decoder.

The Problem

Ecological & Health Impact: HAB cause mass fish mortality, ecosystem disruption, and release toxins harmful to humans and wildlife.
Economic Burden: The 2017-2018 Florida Red Tide alone caused an estimated $2.7 million in losses, with billions lost annually worldwide.
Monitoring Limitations: Existing methods rely on manual water sampling and microscopy—costly, time-consuming, and geographically constrained.
Fragmented AI Solutions: Prior work addresses either bloom severity prediction or spatial segmentation in isolation, limiting comprehensive monitoring capabilities.

Our Solution: ALGOS

ALGOS is a unified vision-language framework that bridges reasoning segmentation with HAB severity assessment in satellite imagery.

Dataset Curation

HAB Segmentation Dataset: GeoSAM-assisted annotation with human evaluation on CAML Sentinel-2 imagery. Interactive mask generation with positive/negative prompts and ROI boxes, followed by morphological filtering and human validation.
HAB Reasoning Dataset: Five-level severity scale based on WHO recreational guidance (Level 1: <2×10⁴ to Level 5: ≥1×10⁷ cells/mL). Natural language query templates paired with satellite images and severity labels.

Architecture

Vision Encoder: Remote-CLIP ViT-L/14 optimized for satellite imagery
Language Model: Vicuna-7B with specialized <SEG> token for segmentation
Decoder: SAM decoder head for pixel-level mask prediction
Training: Joint optimization with text generation loss and segmentation loss (BCE + DICE)

Implementation

Trained on 8× NVIDIA DGX A100 (80GB) GPUs
LoRA adapters for efficient fine-tuning
Joint training on HAB, FP-Ref-COCO, and ReasonSeg datasets
~6 hours training time

Figure 2: Interactive segmentation workflow used during data curation. (a) User-provided prompts (green: positive, red: negative), (b) Generated mask overlay, (c) Bounding boxes extracted.

Results

ALGOS achieves strong performance across both segmentation and severity prediction tasks:

Segmentation Performance

cIoU (per-image): 0.6493 (vs. LISAT: 0.1083, LISA-7B: 0.1373)
gIoU (dataset-level): 0.5969 (vs. LISAT: 0.1052, LISA-7B: 0.1274)
ALGOS far surpasses baseline models in accurately capturing both per-image bloom regions and large contiguous extents

Severity Prediction

MSE: 2.984 (vs. LLaVA-7B: 3.868) — 22.8% reduction
RMSE: 1.727 (vs. LLaVA-7B: 1.967)
MAE: 1.365 (vs. LLaVA-7B: 1.587)

Table 1: Segmentation Results — **Table 1:** Segmentation results comparing LISAT, LISA, and ALGOS.

Table 2: Severity Prediction Results — **Table 2:** Severity prediction results comparing LLaVA baseline and ALGOS.

Key Contributions

Unified Framework: First system to jointly perform spatial segmentation and severity-level estimation for HAB monitoring in satellite imagery.
High-Quality Dataset: Semi-supervised curation pipeline combining GeoSAM automation with human validation for robust segmentation masks.
Strong Performance: Significant improvements over baselines in both segmentation (6× cIoU increase) and severity prediction (23% MSE reduction).
Scalable Monitoring: Enables automated, wide-area HAB monitoring for ecological assessment and public health decision-making.

Limitations & Future Work

Geographic Scope: Current evaluation limited to CAML dataset regions and seasons. Generalization to diverse aquatic environments requires larger-scale, cross-region benchmarks.
Data Dependencies: Reliance on curated datasets highlights the need for continuous data integration pipelines that adapt to evolving ecological conditions.
Future Directions: Extending evaluations to global waterbodies, incorporating temporal dynamics, and developing real-time monitoring capabilities for operational deployment.

BibTeX

@article{hsieh2025segthehab, title = {Seg the HAB: Language-Guided Geospatial Algae Bloom Reasoning and Segmentation}, author = {Patterson Hsieh and Jerry Yeh and Mao-Chi He and Wen-Han Hsieh and Elvis Hsieh}, journal = {NeurIPS 2025 Workshop on Tackling Climate Change with Machine Learning}, year = {2025}, url = {https://arxiv.org/abs/2510.18751} }