Resilient Modeling with AWS Batch

resiliencemodelingarchitectureaws batch

The problem in modeling

Scientific modeling work doesn’t look like normal web traffic. The modeler configures a model, kicks off a run, then waits — and the resources that run needs can swing wildly from the last one. A small run might need 2 vCPUs for five minutes; a big one might need 32 vCPUs for two days. You usually can’t tell which it’ll be until the model is fully configured.

That unpredictability breaks the usual ways of running compute. If you provision a server big enough for the worst case, it sits mostly idle and you’re paying for capacity nobody’s using. If instead you run every model on one long-lived server, that server becomes a single point of failure — if it crashes or gets restarted mid-run, the run is just gone, and whoever kicked it off is staring at a stalled job with no idea what happened.

So the real problem is: how do you run jobs whose resource needs and runtimes vary wildly, without dedicating a server to them full-time, and without losing a run when the machine underneath it disappears?

The pattern: three planes, kept apart

To solve this problem, split the system into three pieces that never share responsibilities. The architecture of this solution:

flowchart TD
    U[Web App] --> P

    subgraph CP[Control Plane]
        P[App / Orchestrator]
    end

    subgraph COMP[Compute Plane]
        D[Ephemeral Compute Job]
    end

    subgraph DATA[Data Plane]
        B1[(Config & Metadata)]
        C1[(Input Data)]
        E1[(Logs & Results)]
    end

    P -->|submit job| D
    P -->|read / write| B1
    D -->|read inputs| C1
    D -->|write logs & results| E1

    classDef planeStyle stroke-dasharray: 5 5
    class CP,COMP,DATA planeStyle
  • Control plane — the always-on application. It takes requests, tracks each job’s state, and decides when to submit work — but it never runs the model itself. It’s a small, cheap web server that stays up all the time; the expensive compute (the vCPUs and memory a simulation actually needs) only exists separately, on demand, in the compute plane below.
  • Compute plane — where the model actually runs. A job is submitted, a machine is created for it, the machine runs the model, and once the job succeeds or fails, the machine is destroyed. Nothing about the model’s progress lives on that machine — so if it dies mid-run, there’s nothing to recover. We just submit the same job again on a fresh machine.
  • Data plane — everything that actually matters about a job: its configuration, its input data, its logs, its results. All of it sits in durable storage, separate from both the control plane and the compute plane, so none of it disappears when either one does. That’s what makes recovery simple — since the important stuff was never on the machine that failed, there’s nothing to lose when we resubmit.

This is the architecture as a whole: a web app that’s always reachable, compute that only exists while there’s work to do, and storage that survives both. That split is what makes resilience nearly free — since the compute plane never holds anything you’d miss, recovering from a failure just means resubmitting the same job against the same durable inputs. It’s also what makes the system elastic, since you’re never paying for machines sitting idle.

None of this is tied to one specific technology — the compute plane could be Kubernetes Jobs, Nomad, or a fleet you run yourself. I built this architecture on top of AWS RDS, EFS/S3, and Batch, and that’s the implementation the rest of this piece walks through.

Building it: Engage

I apply this architecture solution to Engage, an open-source platform for energy planning. Engage handles configuring, solving, and tracking energy models end to end.

  • Control plane — Django, running all the time. Queues model builds to Celery over Redis, submits to Batch on Run, and polls Batch on a schedule for status.
  • Compute plane — AWS Batch. Jobs are sized into small/medium/large tiers from the model config. A failed job just resubmits on fresh compute.
  • Data plane — config and job metadata in RDS Postgres, inputs and results in EFS/S3, logs in CloudWatch.

On top of this architecture, Engage becomes a highly accessible, flexible tool for rapid multi-sectoral scenario exploration. Its shared, cloud-based data model, intuitive interface, and visualization capabilities let diverse stakeholder groups, teams, and experts collaborate on energy systems ranging from a single microgrid to national scale.

Summary

In this Work, we solved resilient modeling by splitting the system into three planes — an always-on app, ephemeral compute, and durable storage — built on AWS RDS, EFS/S3, and Batch. A failed job just resubmits, and compute costs nothing when idle. If you’re facing something similar, hope this architecture helps.