Foundation Models

integrate data from various modalities

• Train an object detection model to provide the bounding boxes.
• Use the bounding boxes as prompt for SAM.
Object detection

Original image Ground truth Bounding box prediction Prediction
Foundation Models for Cell Segmentation Foundation Models for Structure Biology • Self-supervised learning models trained on vast amounts of biological data • Protein sequence database (~1 billion of unlabeled sequences) • Protein structure database ( ~200k experimental structures) • Representation learning: capture fundamental properties and patterns   Goal: Develop foundation models on exascale computers to predict disruption events.
Real-time models are being developed and trained to predict the disruption and take appropriate action.• Domain Generalization (DG) aims to learn a generalizable model trained on observed source domains, and directly applied on the target domain which is unseen during training.

Foundation Models for Fusion Energy
Source domain Target domain • Visual-language model (VLM) are trained on massive datasets, like CLIP [1] model, trained on 400 million pairs of images and texts.The diverse data that help VLMs demonstrate impressive zero-shot ability.

Generalizability of Foundation Models
Are the large visual language models good and good enough for generalizability?
We evaluated the model's generalizability experiments.Even without fine-tuning, CLIP outperforms trained ResNet50 by 6.3%.
Although CLIP has shown impressive performance, there is still some scope for improvement.

Generalizability of Foundation Models
Goal: • Develop a computational framework for plant disease detection, surveillance, and prediction.
• Develop the concept of Digital Twins to connect physical and digital representations of plant diseases, enabling timely decision-making and scalable exploration of disease management strategies.

Challenges:
• Various modalities and scales to accurately detect and predict plant diseases.

LLM-based agent
An LLM-based agent is a comprehensive framework consisting of planning, tools, and memory.The LLM acts as a central controller that provides planning and calls external tools to solve complex tasks.

Limitations for existing LLM-based
• Most existing tools uses a single LLM agent.
• Existing tools are not for scientific applications.
• Existing tools usually encounter reliability concerns.Thank you!
a 1.5B parameter GPT-like foundation model using ~10 TB training data collected from the DIII-D tokamak Fusion Reactor.• Data fusion of 2D ECEi data along with 0-d scalar and 1-d profile diagnostic data Foundation Models for Fusion Energy Science Accomplishment and Impact • Run the foundation model on Frontier cluster at ORNL with good scalability • Improve predictive accuracy for disruption forecasting over previous ML tools Scaling of model training on Frontier cluster at ORNL with 16,384 GPUs.Achieved 0.4 fp16 ExaFlops.
Models CLIPCEIL • CLIPCEIL (Boosting Domain Generalization with CLIP by Channel rEfinement and Image-text aLignment) introduces a lightweight module to fine-tune the CLIP and employ the multi-scale visual features.• We proposed channel refinement module to ensure each channel contains domain-invariant (minimizing domain variance) and class-relevant (maximizing class variance) information.• To align the image and text, we maximize the image-text similarity and calculate direction loss using text class descriptions based on data pairs from different classes and domains.
images (large scale) • Plant pictures (medium scale) • live-cell images from Optical microscopy (small scale) • Electronic microscopy (micro-scale) • Genomic sequence Multi-modal Foundation Models are utilized to align and integrate data from various modalities into a common latent space.

•Figure 1 Figure 2 .
Figure 1 The overall structure of existing LLM-based agents.

ESM-IF (Inverse Folding): Predict protein sequence based on backbone structure
8 Foundation

Models in Structure Biology ESMBind: Protein-metal ion binding prediction
AUPRC SOTA performance on common metal ions in protein Foundation Models in Structure Biology 9 Task: Predict Disruptions in Fusion Device Challenge: Disruptions have proven very difficult to predict with classic simulation tools, especially in real time.