--- title: "Production Backend: Scaling, Monitoring & Reliability" description: "Run production backend systems: scaling patterns, monitoring, error handling, logging, resilience, capacity planning, and incident response." author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/quizzes/post/production-backend-scaling-quiz --- # Production Backend: Scaling, Monitoring & Reliability Welcome to the "Production Backend: Scaling, Monitoring & Reliability" quiz! This quiz tests your knowledge of key concepts and best practices for running production backend systems at scale. You'll be challenged on topics like scaling patterns, monitoring strategies, error handling, resilience techniques, capacity planning, and incident response. Each question includes detailed explanations to help you learn from any mistakes. Good luck! ## Questions ### 1. What is horizontal scaling? - Add more server instances to distribute incoming load - Upgrade existing hardware with more CPU and RAM resources - Optimize application code to improve per-request efficiency - Migrate local storage to a centralized cloud-based system **Hint:** Think about many servers. ### 2. What is connection pooling? - Maintain a cache of open database connections for reuse - Create a new database connection for every incoming user - Synchronize multiple database instances for high availability - Encrypt all traffic between the application and database **Hint:** Think about reusing database connections. ### 3. What is a message queue? - Buffer tasks between services for asynchronous processing - Sort database records based on their arrival timestamp - Manage the priority of incoming HTTP requests at the edge - Verify the identity of users before they access a service **Hint:** Think about asynchronous processing. ### 4. What is rate limiting in production? - Restrict request volume per user to prevent system abuse - Maximize the data throughput of a specific network interface - Calculate the average response time for all API endpoints - Allocate specific CPU time to different background processes **Hint:** Think about protecting system. ### 5. What is the circuit breaker pattern? - Prevent calls to a failing service to allow it to recover - Retry every failed request immediately until it succeeds - Distribute requests across multiple healthy server nodes - Log every service error into a centralized storage system **Hint:** Think about handling failing services. ### 6. What is an SLA (Service Level Agreement)? - External contract defining uptime goals like 99.9% availability - Internal metric used by developers to track system performance - Software license agreement governing the use of open-source tools - Security protocol used to authenticate third-party API providers **Hint:** Think about availability commitment. ### 7. What is an SLO (Service Level Objective)? - Internal performance target used to guide system reliability - The actual measured availability of a service over 30 days - Legally binding document that specifies financial penalties - Maximum number of concurrent users a single instance supports **Hint:** Think about internal performance target. ### 8. What is an error budget? - Calculated allowable downtime used to prioritize new changes - Financial allocation for repairing production infrastructure - Total number of bugs allowed in a release during QA testing - Threshold of failed requests before an alert is triggered **Hint:** Think about acceptable downtime. ### 9. What is incident response? - Structured process for detecting and resolving service outages - Automated scaling of resources during a sudden traffic spike - Customer support workflow for handling user feature requests - The initial planning phase of a software development lifecycle **Hint:** Think about handling outages. ### 10. What is MTTR (Mean Time To Recover)? - Average time required to fix a service after an issue starts - Average time between two consecutive system failure events - Total time spent on developing new reliability features - Minimum time a server can run before a required reboot **Hint:** Think about recovery speed. ### 11. What is chaos engineering? - Inject intentional system failures to test resilience - Run production systems without any monitoring or logs - Randomly update dependencies to find potential bugs - Allowing developers to push code without a review process **Hint:** Think about intentional failures. ### 12. What is graceful degradation? - Turn off non-core features to keep the primary service live - Permanently remove old features to simplify the codebase - Automatically restart servers when they reach high latency - Reduce the security requirements for internal service traffic **Hint:** Think about reduced functionality under load. ### 13. What is bulkhead isolation? - Partition system resources to prevent total failure spread - Replicate data across different physical disk drives - Encrypt data stored at rest to prevent unauthorized access - Use a single large server to handle all service components **Hint:** Think about fault isolation. ### 14. What is a canary deployment? - Release new code to a small group of users before everyone - Deploy the new version to all production servers at once - Run the old and new versions in a simulated environment - Switch all traffic from the old cluster to the new cluster **Hint:** Think about risky rollouts. ### 15. What is structured logging? - Use JSON or key-value pairs to make logs machine-parsable - Write logs in chronological order within a text file - Categorize logs into simple levels like INFO, WARN, and ERROR - Manually review every log entry at the end of the day **Hint:** Think about machine-readable logs. ### 16. What is log aggregation? - Collect logs from multiple sources into a central search tool - Delete old logs automatically to save on storage costs - Convert raw logs into a summarized PDF for management - Compress log files before they are sent to long-term storage **Hint:** Think about collecting logs from many services. ### 17. What is distributed tracing? - Track a request as it flows through various microservices - Map the physical location of servers across the globe - Monitor the bandwidth usage of all internal load balancers - Follow the career path of developers across different teams **Hint:** Think about tracking requests across services. ### 18. What is a liveness vs readiness probe (Kubernetes)? - Liveness ensures the app runs; Readiness ensures it can take traffic - Liveness tracks CPU usage; Readiness tracks memory usage - Liveness is for internal users; Readiness is for external users - Liveness measures latency; Readiness measures throughput **Hint:** Think about health checks. ### 19. What is a load balancer algorithm? - Method like Round-robin used to distribute traffic to nodes - Encryption process that protects data between the LB and user - Decision tree used to determine if a request should be blocked - Compression routine used to shrink the size of HTTP headers **Hint:** Think about distributing requests. ### 20. What is a database read replica? - Copy of a database used to handle query traffic and offload primary - Complete backup of the database stored in a different region - Temporary table used to store intermediate calculation results - Write-only instance used to record application audit logs **Hint:** Think about scaling read traffic. ### 21. What is an auto-scaling policy? - Set of rules to add or remove instances based on live metrics - Manual schedule used to reboot servers every weekend - Strategy for increasing the pricing tier of a cloud account - Policy that limits the maximum bandwidth a user can consume **Hint:** Think about automatic capacity adjustment. ### 22. What is backpressure handling? - Signal the data source to slow down when the receiver is full - Increase the frequency of requests to bypass slow network nodes - Switch to a different server when the current one becomes slow - Store all incoming data in a permanent database indefinitely **Hint:** Think about when downstreams slow down. ### 23. What impacts cost in scaling? - Usage of compute, data transfer, storage, and redundant nodes - Total number of lines of code written by the engineering team - Frequency of code commits made to the main production branch - Complexity of the CSS used in the frontend user interface **Hint:** Think about infrastructure expenses. ### 24. What is blue-green deployment? - Maintain two identical environments and switch traffic between them - Color-code different versions of the API for developer testing - Deploying code updates on Tuesdays and Thursdays only - Splitting the database into two halves based on the user ID **Hint:** Think about zero-downtime releases. ### 25. What is a feature flag? - Remote toggle to enable or disable code without a new deploy - Special icon used in the UI to highlight new application updates - Code comment that marks a section for future development - Strict rule that prevents any new features from being added **Hint:** Think about gradual rollout control. ### 26. What is observability vs monitoring? - Monitoring alerts on symptoms; Observability enables deep debugging - Monitoring is for hardware; Observability is for software - Monitoring is automatic; Observability requires manual data entry - Monitoring uses logs; Observability only uses metrics and traces **Hint:** Think about understanding systems. ### 27. What is an alerting strategy? - Notify on actionable symptoms instead of noisy low-level events - Send an email to the whole company for every server warning - Only trigger alerts during standard business hours (9am to 5pm) - Log every alert to a local text file without sending notifications **Hint:** Think about notifying about problems. ### 28. What is eventual consistency? - Distributed data model where replicas will eventually align - Requirement that all database writes must succeed or fail together - System state where data is deleted after it becomes one year old - Configuration that ensures every server has the exact same hardware **Hint:** Think about data across replicas. ### 29. What is a cache invalidation strategy? - Logic used to decide when to remove or refresh cached data - Process of adding more RAM to the caching server cluster - Technique for encrypting cache keys to prevent data leakage - Method for ensuring that every user gets a unique cache key **Hint:** Think about stale data. ### 30. What is security hardening in production? - Apply multiple layers of protection to minimize the attack surface - Physically lock the server racks in a secure data center - Require all developers to change their passwords every 24 hours - Remove all external internet access from the entire company network **Hint:** Think about protecting systems.