Integrate with Modal¶
Run Stardag tasks on Modal's serverless infrastructure.
Overview¶
Modal provides serverless cloud computing for engineers who want to build compute-intensive applications without managing infrastructure. The Stardag Modal integration enables:
- Serverless execution of tasks
- Automatic scaling
- Flexible routing of individual tasks to appropriate compute resources, including GPU access
Prerequisites¶
Modal Account¶
- Sign up for a Modal account.
- Optionally create a new dedicated Modal environment, or stick with the default
mainenvironment.
Stardag Registry Environment (Optional)¶
We recommend setting up the Stardag Registry.
You can also run Stardag on Modal, completely without the Registry.
Sign up at app.stardag.com or follow the setup guide for running it self-hosted.
You're all set. Just skip using a Stardag API-key in the examples.
Minimal Example from Scratch¶
We are going to create a new minimal Python project with the following structure:
Create and install the project¶
Create the new project (with uv as build system):
mkdir stardag-modal
cd stardag-modal
cat > pyproject.toml << 'EOF'
[project]
name = "stardag_modal"
version = "0.0.1"
requires-python = ">=3.12"
dependencies = ["stardag[modal]>=0.1.2", "modal"]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
EOF
mkdir stardag_modal
touch stardag_modal/__init__.py
touch stardag_modal/main.py
And install it:
Now in stardag_modal/main.py let's define some minimal tasks that we can compose into a DAG:
# stardag_modal/main.py
import sys
import modal
import stardag as sd
import stardag.integration.modal as sd_modal
@sd.task(name="Range")
def get_range(limit: int) -> list[int]:
return list(range(limit))
@sd.task(name="Sum")
def get_sum(integers: sd.Depends[list[int]]) -> int:
return sum(integers)
Then let's define the modal image we will be using:
# stardag_modal/main.py continued...
# Must match local Python version for Modal serialization compatibility
python_version = f"{sys.version_info.major}.{sys.version_info.minor}"
# Define the Modal image
image = (
modal.Image.debian_slim(python_version=python_version)
.uv_sync()
.add_local_python_source("stardag_modal")
)
# Define the StardagApp. The Stardag Registry API key is injected into
# every function automatically from the `stardag-api-key` Modal secret
# (created below via `stardag modal stardag-api-key create`); see the
# `stardag_api_key_secret` argument to override the name/secret or set
# it to None if you supply the key another way.
app = sd_modal.StardagApp(
"stardag-poc",
builder_settings=sd_modal.FunctionSettings(image=image),
worker_settings={
"default": sd_modal.FunctionSettings(image=image),
},
)
# stardag_modal/main.py continued...
# Must match local Python version for Modal serialization compatibility
python_version = f"{sys.version_info.major}.{sys.version_info.minor}"
# Define the Modal image
image = (
modal.Image.debian_slim(python_version=python_version)
.uv_sync()
.add_local_python_source("stardag_modal")
)
# Define the StardagApp
app = sd_modal.StardagApp(
"stardag-poc",
builder_settings=sd_modal.FunctionSettings(image=image),
worker_settings={
"default": sd_modal.FunctionSettings(image=image),
},
)
And finally, compose the tasks and add a main section for building them on modal:
# stardag_modal/main.py continued...
root_task = get_sum(integers=get_range(limit=21))
if __name__ == "__main__":
res = app.build_spawn(root_task)
print(res)
Now that we have the code in place and the stardag and modal Python packages installed, we need to set up the environment before we can run the example.
Set up your Modal environment¶
Authenticate with modal (if you haven't already):
If you've created and want to use a dedicated Modal environment, make sure to also set:
Set up your Stardag environment¶
When running Stardag on Modal, we must use a remote filesystem for our target roots. A natural choice when running on Modal is to use Modal volumes:
Create a new isolated Stardag environment:
Add and activate a new profile for the environment:
We also need to give modal functions access to the Stardag Registry:
Deploy the app¶
Now let's deploy the app to Modal.
You should see output like:
Using active stardag profile
Registry URL: https://api.stardag.com
Workspace ID: <ws-id>
Environment ID: <env-id>
Target roots:
default: modalvol://stardag-poc/target-roots/default
Modal volumes:
default: stardag-poc
Functions:
build
worker_default
✓ Created objects.
├── 🔨 Created mount PythonPackage:stardag_modal
├── 🔨 Created mount PythonPackage:stardag
├── 🔨 Created function build.
└── 🔨 Created function worker_default.
✓ App deployed in 2.592s! 🎉
View Deployment: https://modal.com/apps/<modal-user>/<modal-env>/deployed/stardag-poc
You can also navigate to your modal apps in the relevant environment and should see:
Run the app¶
Now let's execute the main.py module:
Then navigate to the app in the Modal UI to follow the execution progress.
Inspect the results¶
The easiest way to get the results is to use an instance of the desired task and load its output.
Output:
You can also "tab" your way through the DAG dependencies to access root_task.integers:
If you connected to the Stardag Registry, you can also click the latest build to inspect the DAG execution.
Restart-safe triggering with build_trigger (recommended with Registry)¶
With build_spawn, the registry build id is created inside the Modal build
container — if that container is restarted (preemption, timeout, a Modal-level
retry), the new invocation starts a new build in the registry.
When you use the Stardag Registry, prefer build_trigger: it creates the
build in the registry from the calling process first, then passes the build id
to the build function as resume_build_id. Any restart of the build function
then resumes the same build — tasks whose outputs already exist are detected
during discovery and skipped:
result = app.build_trigger(root_task)
print(result.build_id) # registry build id, minted at the trigger point
result.function_call.get() # optionally block on the Modal build function
Re-triggering with the same build id re-attaches to the build (e.g. after a failure, or to bring a preempted build back up):
To let Modal restart the build function automatically after infrastructure failures (and thereby auto-resume the build), configure retries on the builder:
app = sd_modal.StardagApp(
"stardag-poc",
builder_settings=sd_modal.FunctionSettings(image=image, retries=2),
worker_settings={"default": sd_modal.FunctionSettings(image=image)},
)
Note that build_trigger requires registry credentials in the calling process
(the active stardag profile), in addition to Modal credentials — unlike
build_spawn, which only needs Modal credentials locally.
Detached execution: running tasks survive restarts¶
Tasks are executed as detached Modal function calls by default: the worker
invocation is spawned (not held open by a blocking call), and its function
call id is recorded in the registry with the task's started event. When a
build is resumed (via resume_build_id / build_trigger), tasks that are
still running in live workers are re-attached instead of re-executed — a
preempted or restarted build function does not restart your long-running
tasks. This also applies across builds: if another build is already running
the same task, the new build attaches to that execution rather than
duplicating it.
Detached mode also makes cancellation real: when a build fails fast or is cancelled, the tracked function calls are explicitly cancelled on Modal (with the legacy blocking mode, workers of a dead build kept running to completion).
Workers additionally report their own lifecycle events (started, completed
with artifacts, suspended, failed) directly to the registry when the app has
registry credentials — so a task's registry state stays accurate even if the
build function dies while the task is running. If you drive a deployed app
built with an older stardag version from a newer local SDK, pass
ModalTaskExecutor(worker_reports_lifecycle=False) (or redeploy the app) so
the build engine doesn't skip events the old workers won't send.
Modal executions also record descriptive executor metadata with task
starts and triggered builds — the Modal app name, workspace, environment,
and function name — which the Stardag UI surfaces (e.g. as deep links to
the Modal dashboard). Resolution is automatic and best-effort (the
workspace comes from a cached Modal token lookup); pass
StardagApp(modal_workspace=...) to set the workspace name explicitly.
Reactive scheduling: no resident build function (experimental)¶
With build_trigger(..., reactive=True) the build runs with no resident
orchestrator at all: task discovery happens at the trigger, and the build
is driven by short-lived scheduler ticks — spawned when the build is
triggered, whenever a worker finishes a task, and (recommended) by a
periodic watchdog. Between ticks, nothing runs except your tasks: a
multi-day build with a few long-running tasks costs no orchestrator
container time, and there is no orchestrator process whose crash could
affect the build.
app = sd_modal.StardagApp(
"stardag-poc",
builder_settings=sd_modal.FunctionSettings(image=image),
worker_settings={"default": sd_modal.FunctionSettings(image=image)},
# Recommended with reactive mode: periodically re-check running builds
# (covers lost wake-ups and builds cancelled from the UI).
watchdog_period_minutes=5,
)
# After deploy:
result = app.build_trigger(root_task, reactive=True)
# Re-trigger with the same build id to wake a stalled build, to add new
# root tasks to the running build, or to change the tick configuration:
app.build_trigger(
more_tasks, build_id=result.build_id, reactive=True,
tick_kwargs={"linger_seconds": 60},
)
Requirements and current limitations: the app must be deployed with this
stardag version — both the Modal app and the registry server (an older
server fails reactive triggers with a "does not support reactive
scheduling" error); the triggering process needs registry credentials and
access to the default target root (task objects are persisted there as
pickles for the ticks — the reactive marker, owning app, and tick config
live in the registry, not on the target root, so re-triggering works even
when the target root is immutable/append-only and a re-trigger may update
tick_kwargs); the global concurrency lock and build-local
ConcurrencyConfig limits are not applied by ticks (use the
registry-backed named limits above; Modal's per-function
concurrency_limit also still applies). Builds cancelled from the
registry UI are picked up by the next tick (within the watchdog period),
which cancels the running Modal function calls; on failure, tasks
transitively blocked by the failed task are marked skipped.
Two operational notes:
- Avoid redeploying the app with changed task definitions while
reactive builds are in flight. Task objects are persisted as pickles
for the scheduler; if a stored pickle becomes unloadable (e.g. after a
redeploy), the tick falls back to reconstructing the task from the
registry's stored data — which works as long as the task class is
still importable and its fields are compatible (nested task fields
must use
sd.TaskLoads/sd.SubClassannotations). Only if both paths fail is the task failed by the next tick (never a silent stall). - The watchdog sweep runs one quick scheduling pass per running build (it skips the linger), so its per-period cost is one short function invocation plus a frontier query per running build.
- Define the callables you pass to
StardagAppin an importable module.worker_selector,limit_key_selector, and any custom build/run functions are captured by the serialized Modal functions (build, workers, and the reactivetick/tick_watchdog), which Modal deserializes in fresh containers — including the scheduled watchdog, which always runs cold. A plain module-level function is pickled by reference, so its defining module must be importable in the container: put these callables in a module that is part of the source you add viaadd_local_python_source(...), not in a loose deploy script. A selector defined directly in a script deployed by path (stardag modal deploy app.py, which Modal loads as a top-level module namedapp) fails to deserialize on the first cold container with aModuleNotFoundErrorfor theappmodule. (Themodal/basicexample is unaffected only because it passes no such callables; themodal/walkthroughexample keeps its selectors in a dedicatedselectors.pyfor exactly this reason.) - Every deployed function needs the registry secret — the workers
self-report their lifecycle (started/completed/…) and the tick/watchdog
read and update build state, so all of them make registry calls and
401without credentials. As of stardag 0.10.2 this is handled byStardagApp(stardag_api_key_secret=...): the named secret (default"stardag-api-key", created bystardag modal stardag-api-key create) is injected into every function, so you declare it once — or just rely on the default and don't declare a registry secret at all. (On stardag ≤ 0.10.1 you instead had to put the secret on the builder — 0.10.1 — or on every worker — 0.10.0.)
Named concurrency limits are enforced registry-side in reactive mode —
across builds, not just within one. Configure caps per environment
(PUT /api/v1/concurrency-limits/{key} with {"max_concurrent": N}) and
tag tasks with keys on the app (deployed configuration, applied
consistently by every scheduler tick):
app = sd_modal.StardagApp(
"stardag-poc",
builder_settings=sd_modal.FunctionSettings(image=image),
worker_settings={"default": sd_modal.FunctionSettings(image=image)},
watchdog_period_minutes=5,
limit_key_selector=lambda task: ["gpu"] if needs_gpu(task) else [],
)
A task denied by a limit stays pending and is retried when a slot frees (immediately for same-build releases; within the watchdog period for releases in other builds).
When limits are enforced, the watchdog is strongly recommended
(watchdog_period_minutes=5): a slot is freed by the holder reaching a
terminal status, and the watchdog is the safety net that keeps statuses
honest when wake-ups are lost — including the escape hatch that fails a
task stuck RUNNING without an execution ref (default after 30 minutes,
TickConfig.stale_running_no_ref_seconds), which would otherwise hold
its slots indefinitely. Also note that limit-key tags recorded at a
task's start persist until its next start with keys — a later build
re-running the same task id without tags briefly counts under the old
keys while RUNNING.
Server requirement: concurrency-limit enforcement (like reactive mode as a whole) needs a stardag-api version matching this SDK — an older server silently ignores the enforcement parameters, so upgrade the server before relying on limits.
App ownership. Each reactive build is owned by the StardagApp
that triggered it (app_name recorded in the build's reactive metadata in
the registry, read by every tick from the build frontier). With
several apps deployed in one environment, every watchdog sweeps all
running reactive builds — but a tick from a non-owning app never drives
the build with its own commit's code and selectors (or unpickles the
owner's task store, which may not match its code). Instead it
forwards: it spawns the owner app's tick (best-effort) and returns
outcome="foreign_app" — so wake-ups that land on the wrong app are
not lost, and every app's watchdog doubles as cross-app coverage (the
owner-side scheduler lease collapses duplicate forwards). Redeploying
the same app name is the normal upgrade path and unaffected.
To migrate a build to a different app, re-trigger it from that app
(build_trigger(tasks, reactive=True, build_id=<existing id>)): the
re-trigger updates the reactive metadata (owning app + tick config) in the
registry and re-persists the task objects under the new app's code. Two
handoff details: ownership takes effect for new
ticks — a tick of the previous owner that is mid-linger keeps driving
the build until its linger deadline passes (bounded by its
linger_seconds); and wake-ups from the previous owner's still-running
workers reach the new owner via the forwarding above. Symptom worth
knowing: a build not progressing while tick logs show foreign_app
with failed forwards means the owning app was deleted — the build
is orphaned; re-trigger it from a live app to adopt it.
The same named limits can be enforced from resident (non-reactive)
builds via stardag.build.RegistryConcurrencyLimiter — both modes share
the slots. Two caveats when mixing modes: a crashed resident build has
no automatic healer (its RUNNING task holds the slot until explicitly
failed/cancelled via the API/UI — the worker-reporting/tick self-healing
story above is reactive-only), and a legitimately long-running ref-less
resident task that also appears in a concurrently ticking reactive build
can be force-failed by that build's stale_running_no_ref_seconds
escape hatch — raise the bound if you mix modes over the same long
tasks.
Requirements and current limitations: the app must be deployed with this
stardag version (scheduler tick function + self-reporting workers); the
triggering process needs registry credentials and access to the default
target root (task objects are persisted there for the ticks — the
reactive marker/owner/tick config live in the registry); the global
concurrency lock and build-local ConcurrencyConfig limits are not applied
by ticks (use the registry-backed named limits above; Modal's per-function
concurrency_limit also still applies). Builds cancelled from the registry UI are picked up by
the next tick (within the watchdog period), which cancels the running
Modal function calls.
To opt out (legacy blocking remote calls), pass detached=False:
app = sd_modal.StardagApp(
"stardag-poc",
build_function=sd_modal.Builder(detached=False),
builder_settings=sd_modal.FunctionSettings(image=image),
worker_settings={"default": sd_modal.FunctionSettings(image=image)},
)
See Also¶
- Stardag Modal Examples - Ready-to-run Modal examples in the
stardag-examplespackage. - Modal Documentation - Modal features
- ML Pipeline Example - Complete ML pipeline walkthrough
- Integrate with Prefect - Prefect orchestration