SecOps, Disaster Recovery & SOC 2

Security Is a Strategy Function

Most engineers meet security as friction: the MFA prompt, the access request that takes a day, the deploy that needs a review. A CTO has to see it from the other side of the table, where it looks like two things: a sales enabler and existential risk management. Neither of those is delegable to "the IT person," because both are bets the whole company is making.

The sales-enabler part is concrete. The moment Hireboard sells to a company with a procurement department, the deal routes through a security review: a 200-400 question questionnaire, a request for your SOC 2 report, and a security architect on the buyer's side whose job is to find a reason to say no. Deals rarely die there loudly — they die by stalling. "Waiting on the security review" is where six-figure contracts age out of the quarter. A company that answers the questionnaire in two days from standing documents, attaches a current Type II report, and speaks fluently about incident response closes deals its competitors stall on. Security posture is, quite literally, sales velocity.

The existential part is equally concrete. A startup survives bad quarters and failed features; very few survive telling every customer their data is gone, or that an attacker had it for three months. The post-incident graveyard is full of companies that were fine on Monday: a deleted production database with backups nobody had ever restored, ransomware that encrypted the backups too, an offboarded contractor whose credentials still worked. None of these are exotic attacks — they are operational sloppiness compounding until probability catches up.

So the posture to build is deliberately unheroic: boring, documented, rehearsed. Boring means the controls are defaults, not per-incident decisions — SSO on everything, least privilege by construction, backups nobody has to remember to run. Documented means the policies exist, are short, and match reality. Rehearsed means you have actually restored a backup, actually run an incident drill, and actually know how long recovery takes — before the day it matters. Everything in this module is in service of those three words.

Disaster Recovery: Writing a Plan That Works

A disaster recovery plan is not a compliance artifact. It is the document your most junior on-call engineer follows at 3 a.m. while you are on a plane. Write it for that reader, in that moment. The sections below are the plan, in order.

Step 1: Define RTO and RPO Per System

Two numbers govern everything else. RTO (Recovery Time Objective) is how long the business can tolerate a system being down. RPO (Recovery Point Objective) is how much data the business can tolerate losing — measured as time between the last recoverable state and the failure. RTO drives your failover architecture; RPO drives your backup frequency.

The critical discipline is setting these per system, in tiers. Uniform targets are unaffordable: giving your analytics warehouse the same one-hour RTO as your payments ledger means paying for hot standby infrastructure on a system nobody would notice was down for a day. Conversely, letting your payments system inherit the analytics tier is how you lose money movements. Rank every system, argue about the tiers with the business, and write the result down:

Notice what the table buys you: every architecture and budget conversation about resilience now has a shared vocabulary. When someone proposes multi-region active-active for a Tier 2 system, the table says no for you. When finance asks why the database bill has a standby replica in it, the table answers that too.

Step 2: A Backup Strategy That Matches the RPO

Backups are the mechanism that makes RPO real. The rules that matter:

• Frequency follows RPO. A 15-minute RPO means continuous WAL archiving or replication, not a nightly dump. A 24-hour RPO can be a nightly snapshot. If your backup schedule cannot mathematically meet the tier's RPO, one of them is a lie — fix whichever one is wrong.
• The 3-2-1 rule. Three copies of the data, on two different storage systems, one of them offsite. In cloud terms: the live database, snapshots in the same region, and a copy replicated to a different region or a different cloud account. The failure you are defending against is correlated loss — a region outage, or an attacker with your primary account's keys.
• Immutability is non-negotiable. Modern ransomware playbooks delete or encrypt backups first, then the production data, because attackers know backups are your leverage. Use object-lock / write-once storage or a separate account with a deny-delete policy, so that even an admin credential compromised in your primary account cannot destroy the backup history.
• Retention is a policy decision, not a default.Keep dailies for 30 days, weeklies for 90, monthlies for a year — or whatever your data retention policy says. The point is that the number is chosen, written down, and consistent with what you tell customers.
• Back up the things that are not the database.Object storage, configuration, secrets metadata (not the secrets themselves — those live in the secrets manager, which has its own recovery story), DNS zones, and the infrastructure code that rebuilds everything else.

Step 3: Runbooks Per Scenario

Generic plans fail because disasters are specific. Write a runbook for each scenario you consider credible, and keep each one to a page or two of numbered steps. Every runbook names an owner, a backup owner, and the communication tree: who declares the incident, who updates the status page, who calls the customers that have contractual notification clauses. The minimum scenario set:

Step 4: The Test Discipline

An untested backup is a hope, not a backup. The industry is littered with teams that ran backups faithfully for years and discovered at restore time that the dumps were empty, the snapshots were of the wrong volume, or the restore took four days against a four-hour RTO. The only proof a backup works is a completed restore, and the only proof your RTO is real is a timed drill. So schedule them:

• Restore drills: on a cadence matching the tier (quarterly for Tier 0, twice a year for Tier 1, annually below that), restore a production backup into an isolated environment, verify row counts and spot-check integrity, and record the wall clock time.
• Game days: chaos engineering lite. Announce a scenario ("the primary region is gone as of 10:00"), have the on-call team execute the runbook against staging, and take notes on every step that was ambiguous, missing, or slower than expected. The runbook edits from a game day are worth more than any amount of review-in-a-document.
• Record actual vs. target: keep a table of drill date, scenario, target RTO/RPO, and actual achieved numbers. This is simultaneously your improvement backlog and the single best artifact to hand an enterprise security reviewer or auditor.

The skeleton every DR plan needs, in order:

Data Integrity: The Rules and the Proof

DR protects you from losing data. Data integrity protects you from something quieter and often worse: keeping data that is wrong. Corrupt balances, orphaned records, and double-charged customers do not page anyone — they accumulate silently until an audit, a customer complaint, or a reconciliation attempt exposes months of drift. Integrity is therefore both a document (the rules) and a practice (the machinery that proves the rules hold). The rules:

• Validate at the boundaries, constrain in the database.Application-level validation is your first line and your best error messages — but the application lies. It has bugs, it gets bypassed by scripts and backfills, and a second service will eventually write to the same tables. Database constraints are the last line that does not lie: foreign keys, unique constraints, not-null, and check constraints encoding real invariants (quantities non-negative, state transitions from a legal set). If an invariant matters, it lives in the schema, not just the code.
• Append-only audit logs for money and state changes.Any operation that moves money, changes an entitlement, or alters a record with legal weight gets an immutable audit entry: who, what, when, before-value, after-value. Updates to the audit table are revoked at the database-permission level. When a customer disputes a charge or an auditor asks how a balance came to be, this table is the answer; without it, the answer is archaeology.
• Reconciliation jobs that prove invariants — and page.For every invariant that matters, write a nightly job that checks it against the whole dataset: the sum of ledger entries equals the sum of account balances; every subscription has exactly one active billing record; every invoice total equals the sum of its lines. When the check fails, it pages — a broken invariant is an incident, because every hour it persists widens the blast radius of whatever caused it. Reconciliation is how you find the bug that validation and constraints both missed, on day one instead of month six.
• Soft delete plus a retention policy. Hard deletes destroy your ability to investigate and to recover from application bugs. Soft-delete by default, then honor your data retention policy (and deletion requests under privacy law) with a scheduled hard-purge job. Deletion becomes a controlled, logged, two-phase operation instead of a foot-gun in every code path.
• Idempotency for money operations. Every payment, refund, and credit API accepts an idempotency key and returns the original result on retry. Networks fail mid-request; without idempotency, every timeout is a potential double charge. This is also a processing-integrity control auditors recognize by name.
• Migration safety: expand, migrate, contract.Never make a destructive schema change in one step. Add the new column or table (expand), backfill and dual-write while both shapes coexist (migrate), and only after verification remove the old shape (contract). Renames, drops, and type changes done in one deploy are the leading self-inflicted cause of the database corruption runbook getting used.

The Document Set a Real Company Needs

At some point — usually triggered by the first enterprise deal or the SOC 2 decision — you need actual written policies. The purpose is not ceremony. Each document answers a question a buyer, auditor, or court will eventually ask, and forces a decision you would otherwise make inconsistently under pressure. The inventory:

The writing rule that governs all of them: short, true, and followed beats long, aspirational, and ignored. A two-page access policy your team actually follows is worth more than a forty-page one copied from a template library. This is not just taste. Auditors test the gap between what the policy says and what practice shows, and every aspirational sentence is a finding waiting to happen. Courts and regulators are harsher still: a policy you wrote and did not follow is evidence that you knew the standard and chose not to meet it. Write only what you will do; then do it.

SOC 2: What It Is and How to Actually Pass

What it is — and is not

SOC 2 is an attestation, not a certification. There is no pass/fail badge issued by a standards body; a licensed CPA firm examines your controls and writes a report attesting to what they found, including any exceptions. Buyers read the report — which is why "SOC 2 compliant" on a website means little and the actual report under NDA means a lot.

The examination is against the Trust Services Criteria. Security (the "common criteria") is mandatory; the other four are optional scope you choose:

Scope only what buyers demand. Every added criterion is more controls, more evidence, and more surface for exceptions. Security plus Availability is the standard first-audit scope for a SaaS company; add others when a contract actually requires them.

Type I vs. Type II

A Type I report says: on this date, controls were suitably designed and in place. A snapshot. A Type IIreport says: over this observation window (three to twelve months), the controls operated effectively — the auditor sampled evidence across the whole period. Buyers know the difference, and what they really want is Type II; a Type I mostly buys you a line in a sales conversation ("Type I complete, Type II in progress, report expected in Q3"). A common pragmatic path is a Type I to satisfy an in-flight deal, with the Type II observation window starting immediately after. If no deal is forcing the issue, skipping straight to Type II saves money and a redundant audit cycle.

The realistic path and timeline

Total for a first Type II: commonly six to twelve months from decision to report in hand. Plan backwards from when sales needs it, and remember the report goes stale — buyers expect a fresh one annually, so the observation window effectively never closes again.

Evidence culture

The audit is not proving you are secure. It is proving you did what you said: that the access review your policy promises quarterly actually happened four times, with a record; that the terminated employee's access was removed the day the policy says, with a ticket showing it. Screenshots, tickets, review sign-offs, and logs are the currency. The design principle that follows: controls must generate their own evidence.Access approved through a ticketing flow leaves a record for free; access granted in a Slack DM means someone reconstructing screenshots at audit time. Offboarding driven by the IdP logs itself; offboarding done by memory does not. Any control that requires a human to remember to document it will become your audit bottleneck — and probably an exception.

Where first audits actually go wrong

• Orphaned access after offboarding. The auditor cross-references the HR termination list against every system's user list. The contractor from March who still has repo access is the single most common finding in first audits.
• Missing or hollow access reviews. Either the quarterly review just did not happen, or it happened as a rubber-stamp with no evidence anyone actually looked. Reviews that never remove anything draw auditor attention.
• Policies staff have never read. The policy says annual acknowledgment; half the team has no acknowledgment on record; two engineers, when interviewed, do not know the incident process exists.
• No tested DR restore. The DR plan promises backups and recovery targets, and there is no record of a restore ever being performed. Everything in the test-discipline section above is also audit evidence.
• Exceptions without documented risk acceptance.Deviating from your own policy is sometimes the right call — but an undocumented deviation is a finding, while the same deviation with a written, ownered, time-boxed risk acceptance is a functioning risk-management program. The paper trail is the difference.

The Habits That Make All of This Cheap

Everything above sounds expensive. It is expensive — if you bolt it on. It is nearly free if a handful of habits are defaults from early on, because then the compliance program is mostly describing what already happens:

• Least privilege by default. Access is granted by role, expanded by request with a reason, and expiring where possible. The question is never "why would we restrict this?" but "why does this role need it?" Quarterly reviews then remove drift instead of excavating years of accumulation.
• SSO + MFA everywhere, from the start. One identity provider in front of every tool means onboarding is a group assignment, offboarding is one deactivation, and the auditor's access questions have one answer. Every tool adopted outside SSO is future offboarding debt.
• Secrets live in a manager, never in code.Secrets in a vault with access logging and rotation; scanning in CI to catch the ones that try to sneak into the repo. A secret in git history is permanently leaked — rotation, not deletion, is the only fix.
• Immutable infrastructure, trending. Servers and configuration built from code and replaced, not patched by hand. Every hand-edit is unauditable drift; infrastructure-as-code makes the change management policy enforce itself, because the change is a reviewed pull request.
• Blameless postmortems. The incident you punish is the incident that goes unreported next time — and the unreported incident is the one that recurs at twice the size. Blameless does not mean consequence-free; it means the analysis targets the system that allowed the mistake, and action items get owners and deadlines.
• Security review inside the change process, not a quarterly ambush. A short checklist on the pull request template — new data collected? new external surface? new vendor? authz changes? — catches issues at the moment they are cheapest to fix, and generates continuous evidence that security review happens. The periodic audit then confirms the process instead of substituting for it.

The compounding effect is the point. A company with these defaults reaches its SOC 2 gap assessment and finds the remediation list short, its security questionnaires answerable from standing documents, and its 3 a.m. incidents survivable by whoever is on call. That is what boring, documented, rehearsed buys: the worst day the company will have is one you have already practiced.

Knowledge Check

Test your judgment on the operational playbook before moving on.