An interactive flow-based programming platform for streaming experimental data to High-Performance Computing (HPC). Users can wire together containerized operators through a web frontend and deploy them on both HPC and edge resources.
demo.mp4
- Interactive Web Frontend: Create data pipelines with a React-based frontend.
- Containerized Operators: Uses container technology to ensure consistent execution environments.
- Data Streaming: Operators connect and data flows between them over network with point-to-point communication.
- Docker Desktop and Podman Desktop (recommended for easiest setup)
- NATS Tools: Command-line utilities for interacting with NATS
-
Set up Environment Variables
Add Github and personal token information in.env
file.GITHUB_USERNAME=your_github_username GITHUB_TOKEN=your_personal_token
-
Generate NATS Authentication Files
Generateauth.conf
for the NATS cluster and various.creds
and.nk
files for NATS authentication.cd nats-conf ./generate-auth-jwt.sh
-
Build the Required Images
Use one of the following methods.docker/build.sh
or
docker/bake.sh
-
Bring up Services
Use docker compose:docker compose up --force-recreate --remove-orphans --build -d
This project is licensed under the LBNL BSD-3 License - see the LICENSE file for details.