Shared Test Environment
Developer can run CI jobs in the test environment and it’s very close to the prod environment. The stability of it is treated with the same respect as trunk which stays open and stable so developer can get a clear signal of their tests by testing their local changes again the shared test environment.
Canary services can be rollout little by little and direct traffic towards it. If there’s any problem, we can roll back the service. Netflix Asgard is a deployment management which manages red/black pushes.
Service dependency visualization should be able to answer questions like, how many dependencies does my services have? What’s the call volume on my service? Are any dependency service running hot? What are the top N slowest “business transaction”? Distributed tracing can be used to build a graph of all the service dependency. Some APM (application performance monitoring) tools like Dyntrace would also be able to trace a request through multiple services. Dashboard metrics (quick scan of services health, median, 99%ile, 99.9%ile) are also essential, e.g. Netflix hystrix / turbine. And 500px is using datadog. Google has cloud trace and cloud debug (part of Google Cloud Monitoring). Some of the other metrix to measure incident response are MTTD (mean time to detection), MTTR (mean time to resolution) and the method of detection.
12 factor application
The twelve-factor app is a methodology for building software-as-a-service apps that:
- Use declarative formats for setup automation, to minimize time and cost for new developers joining the project;
- Have a clean contract with the underlying operating system, offering maximum portability between execution environments;
- Are suitable for deployment on modern cloud platforms, obviating the need for servers and systems administration;
- Minimize divergence between development and production, enabling continuous deployment for maximum agility;
- And can scale up without significant changes to tooling, architecture, or development practices.
Below is the 12 factors:
- Codebase – one codebase tracked in revision control, many deploys
- Dependencies – Explicitly declare and isolate dependencies
- Config – Store config in the environment
- Backing Services – Treat backing services as attached resources
- Build, release, run – strictly separate build and run stages
- Processes – execute the app as one or more stateless processes
- Port binding – export services via port binding
- Concurrency – scale out via the process model
- Disposability – maximize robustness with fast startup and graceful shutdown
- Dev/prod parity – keep development, staging and production as similar as possible
- Logs – treat logs as event streams
- Admin processes – run admin/management tasks as one-off processes
Automate everything, from building, testing to deployment, rollback. Developer should be self-service, no need to ask for production team or building team to enable them to develop.
Each service should have a mock service for the consumer service to do testing. It’s best to be provided by the team who develop the service but in reality, it often developed by consumer service team.
Service Chassis (@Kixeye)
Service chassis is a framework to develop microservice that has all the core integration included.
- Configuration integration
- Registration and discovery
- Monitoring and metrics
- Load-balancing for downstream services
- Failure management for downstream services
- Dev: service template in maven
- Deployment: build pipeline through puppet -> packer -> ami
- NetflixOSS: asgard, hystrix, ribbon + websockets Janus, eureka