Web-scale systems behave as complex, intractable systems that cannot be fully modeled, simulated, or predicted outside of a live environment. Consequently, staging environments are inherently incomplete for understanding system behavior under stress. ## The Architecture of Intractability Web systems are intractable due to four distinct characteristics. First, they contain too many details to be fully described. Second, they change at a high rate, mutating faster than a complete description can be finished. Third, their components function in ways that are partly unknown because they resonate with each other under varying conditions. Fourth, their internal processes are heterogeneous and irregular. ## Staging Limitations Because component interactions change dynamically under load, many system behaviors can only be observed when subjected to real traffic in production. Relying solely on pre-production testing fails to prepare engineering teams for the unexpected failure modes that inevitably occur in live environments. **Cross-Domain Connections**: [[Controlled Failure Injection Uncovers Production System Surprises]] ## Source - [[Fault Injection in Production]] — ACM Queue/John Allspaw, 2012-08-01 — https://queue.acm.org/detail.cfm?id=2353306