Services

Automation & AI Workflows

The work that breaks when someone forgets — provisioning, deploys, certificates, backups — made reliable and invisible.

Automation is the least glamorous thing we do and the one teams notice soonest. It is taking a process that currently lives in somebody's head — the deploy that only works when the right person runs it, the certificate renewed because someone remembered, the report assembled by hand every second Friday — and turning it into something that happens whether anyone is paying attention or not.

The hard part is never the happy path. Standing up a single WordPress site touches a database cluster, a secrets store, a container scheduler, a load balancer, a DNS provider, a certificate authority and a CDN, and any one of them can fail after the others have already succeeded. What separates a script from a system is what happens at step seven of fourteen: whether the work already done was recorded, whether the retry is safe, and whether anybody finds out.

So the bar is not that it runs. It is that it can run twice — against a half-finished state, at three in the morning, with nobody watching — and arrive in the same place.

The kinds of automation we build.

Six shapes. Each is something we shipped and then had to keep running while other people depended on it, which is where the design decisions actually get tested.

Provisioning, and the eight systems behind one call

One request creates a thing that does not exist yet, and the thing is made out of other people's services. The design work is ordering, failure isolation, and recording what already succeeded so a retry can pick up rather than start over.

Creating a site on Vector runs fourteen steps behind a single API call — an Aurora database and user, credentials into Secrets Manager, an ECS task definition and service, a load balancer target group and host rule, health checks, SSH port allocation and multiplexing config, DNS records, then SSL provisioning for each environment. Resources are written back to the site record as they are created, so a failure at step nine leaves something recoverable instead of something orphaned. More on platforms & hosting.

Work that outlives the request

Provisioning a server, importing a database, building an image — none of it fits inside an HTTP request. Once work moves onto a queue the interesting questions are what a worker does with a failure, and how the caller finds out it finished.

Horizon and Redis on Vector, Sidekiq on Helix, and a RabbitMQ agent fleet spanning four regions on TurboHub. Our provisioning agent returns one of four exit codes — it worked, you asked for something wrong, try again shortly, stop trying — so the queue can tell a refusal apart from a bad afternoon instead of retrying both the same way. We have also built a database-backed job queue inside WHMCS, with tests, because there was not one.

Processes that take minutes and involve strangers

Certificates, DNS propagation, third-party provisioning. Work measured in minutes to hours, owned by somebody else's system, and inclined to finish in a different order than you asked for. Request and response is the wrong shape for it; a state machine is the right one.

Vector provisions SSL through a nine-step progression across a certificate authority, a DNS provider and a CDN edge. A background job advances it every fifteen seconds while it is making progress and every two minutes while it is waiting on somebody else, with separate timeouts for certificate issuance and for propagation. Every step is idempotent, so a duplicate job or a restarted worker is harmless rather than a second certificate.

Things that happen on a schedule, or when something else does

The category a team describes as something they just remember to do, right up until the week nobody does. Cron jobs, event handlers, and the notifications that make both visible when they matter and silent when they do not.

An on-call sync we wrote pulls PagerDuty rotations into Slack user groups on a schedule, so an escalation reaches whoever is on call now rather than whoever was on call in March. Vector emits an event for everything that happens to a site and delivers them as signed webhooks with exponential backoff, a delivery history you can query, and scheduled pruning so the log does not become the next problem. Backup dispatch, retention cleanup and repository health checks all run on the same machinery.

Configuration that corrects itself

The automation that stops a fleet drifting apart. Not a script that configures a server once, but a description of what the server should be, applied continuously, so a hand-edited config file gets repaired instead of discovered eighteen months later.

Our hosting nodes are built by an image pipeline from twenty declarative components and then held in shape by twenty-two Systems Manager documents — Apache, PHP, MariaDB, the firewall, mail, log shipping, credential rotation. The AWS estate underneath them is two CDK codebases, and the legacy fleet runs on Puppet across hosting, DNS, virtualisation and database servers with encrypted secrets. A node being rebuildable from source is a different claim from a node being documented.

The busywork around the actual work

The smallest automations, and often the most appreciated. Nobody writes a case study about a reporting script, but a team gets a morning back every sprint and stops resenting the process.

A sprint reporting tool we wrote for ourselves turned a report assembled by hand out of Jira — fifteen to twenty minutes, every time — into a command that answers in seconds. A self-documenting CLI of roughly thirty commands took developer environment setup from several days down to the day somebody started. Deploy pipelines announce themselves in Slack and roll back in one command.

Twice
how many times an automation has to run correctly to count
14 steps
between one API call and a running WordPress site
0 1 2 3
exit codes, so a queue can tell a refusal from a retry
15s / 120s
how often the certificate progression polls, working versus waiting

Three of them, written up in full.

View all

Orchestration behind an API, an agent fleet across four datacenters, and a migration that had to be resumable because it could not be repeated.

Built to be re-run.

Our control plane never opens an SSH connection to a hosting node. It writes an encrypted payload into Parameter Store and asks Systems Manager to run a command; the binary already sitting on the host reads the payload, deletes it, does the work, and posts a signed callback when it terminates. There is no long-lived server credential on the control plane, no daemon to keep alive on every machine in the fleet, and no inbound port that has to be justified to a security review.

Underneath it sits a rule we apply everywhere: an operation that has already happened is not an error. Creating a user who exists and removing a directory that is already gone both exit zero, which is what makes a retry safe by construction rather than by hope. Callbacks are signed and verified in constant time. Every step records what it did before attempting the next one. None of that is clever — it is just the difference between automation you can re-run and automation somebody has to babysit.

What we build with

LaravelHorizonSidekiqRabbitMQRedisGoBashGitHub ActionsAWS CDKSystems ManagerPuppetDeployerDocker

Where AI actually fits.

The newer half of this page, and the half we are most careful about claiming.

We build with AI in the loop — planning, review, the mechanical half of a migration, the first pass at a test suite. It has changed how quickly the boring parts get done. It has not changed whether the hard parts are hard, and we would rather say that than sell the other version.

The work we think is genuinely worth paying for is the prerequisite nobody advertises: making a system an agent can actually drive. Interfaces that are documented rather than discovered, exit codes that mean something specific, a reference generated from the code so it cannot quietly drift, runbooks written to be read by a machine as well as a person. Vector's CLI can register itself as an MCP server for exactly that reason, and the agent-facing instructions it ships are checked in CI against the commands they describe, so they fail the build rather than the user when they fall out of step.

Which is the honest summary of this half. When a team asks for an AI project and the system underneath it has no contract, no error taxonomy and no current documentation, the useful engagement is an automation project with a clearer interface. The AI part gets easy afterwards, and usually turns out to have been the smaller half of the job.

If your team does something by hand every week, or something that only works when the right person remembers, there is a good chance we have already automated a version of it.