## Core Insight When automation is maintained separately from the system it manages, it suffers bit rot and creates perverse organizational incentives. The most functional tools are written by those who use them. Separating automation ownership from system ownership causes three predictable failures: the automation team has no incentive to reduce the service team's tech debt, the service team has no incentive to make their system easy to automate, and product managers unaffected by automation quality always prioritize features. ## The Cluster Turnup Lesson Google's cluster turnup evolved through four phases: 1. **SSH scripts** — quick initial win, became technical debt 2. **Prodtest** — Python unit tests validating real services, with dependency chains 3. **Idempotent fixes** — each test paired with a fix, safe to run every 15 minutes 4. **SOA-based Admin Servers** — RPC-based, ACL-driven, audit-trailed; service owners responsible for their own turnup RPCs The critical failure occurred between phases: when a separate "turnup team" ran automation on behalf of service owners, quality degraded because the automation maintainers weren't the system operators. ## The Principle Automation code, like unit test code, dies when the maintaining team isn't obsessive about keeping it in sync with the codebase it covers. The world changes around the code — new config options, new package names, new devices — and separated automation falls behind. ## Cross-Domain Applications - **DevOps**: "you build it, you run it" is the same principle — operational knowledge must stay with the development team - **Testing**: test suites maintained by a separate QA team suffer the same bit rot as separated automation - **Documentation**: docs maintained apart from code drift out of sync for identical reasons ## Source - [[Site Reliability Engineering - Chapter 7 - The Evolution of Automation at Google|SRE Ch 7: The Evolution of Automation at Google]] by Niall Murphy ## Related Concepts - [[Automation Value Hierarchy in SRE]] - [[Autonomous Systems vs Automated Systems]] - [[SRE 50 Percent Engineering Rule]]