New to DevOps?

Confused about all the terminology and the funky new tools? If so, this 101 will provide you with the much needed knowledge on DevOps.

Have you ever been invited to a meeting where coffee drinking, pizza eating coders with Chef script T-shirts start to drop names like ZooKeeper, Habitat, Mesos, Bamboo and Kitchen CI? And then walked out of the meeting thinking “I’m going to Docker the Kubernetes out of them!” ?

If so, you should spend the next 5 minutes reading my summary notes on DevOps and DevOps toolchain. I guarantee it will increase your IQ score by a minimum of 3 points, help you sound 10 percent smarter and ensure 20 percent more participation in any DevOps-y meetings.

Introduction to DevOps

DevOps is the practice of operations and development engineers participating together through the entire service lifecycle, from the design and development process all the way to production support.

DevOps is also characterized by operations staff making use of many of the same techniques as developers for their systems work.

The three pillars of DevOps are;

  • infrastructure automation
  • continuous delivery
  • reliability engineering

The three building blocks of DevOps are Agile, Lean and SDLC.

There are principles you can use to guide you in taking the core DevOps values and bringing them to life. The most respected set of principles is called The Three Ways

  1. systems thinking: focus on the overall outcome of the entire pipeline in our value chain
  2. amplifying feedback loops: creating, shortening and amplifying feedback loops
  3. culture of continuous experimentation and learning: create an environment of experimentation and learning

The Three Ways provide a practical framework to take the core DevOps values and effectively implement specific processes and tools in alignment with them.

Five most common DevOps methodologies:

  1. People over processes over tools
  2. Continuous Delivery; it’s the practice of coding, testing,and releasing software frequently, in really small batches so that you can improve the overall quality and velocity.
  3. Lean Management (work in small batches, work in progress limits, feedback loops, visualisation)
  4. Visible Ops Change Control
  5. Infrastructure as Code (systems treated like code, checked into source control, reviewed built and tested, managed programmatically)

10 Practices for DevOps success;

  1. Incident Command System (how to respond)
  2. Developers on call
  3. Status pages (informing users)
  4. Blameless post-mortems
  5. Embedded Teams (Ops member in Dev team and vice versa)
  6. The Cloud (API driven way to create and control infrastructure)
  7. Andon Cords (like a cord on a bus, anyone can stop ship if something is not right)
  8. Dependency injection: a software design pattern, where application must be unaware of its external dependencies. Instead, they are passed into the application at run-time. This is very important for well-behaved application in an infrastructure as a code environment
  9. Blue-Green deployment: two identical systems, upgrade one, test, point production traffic to it, if something fails, point production traffic back to the original system
  10. Chaos monkey: focus on making the overall system highly available

Value Stream Mapping: where you analyse the entire pathway of value creation, and understand what exact value’s added where, how long it takes, and where waste resides in that pathway

In a container-based architecture, the server becomes less and less a part of the equation. Applications are packaged in a container with just enough OS and dependencies to support them, and then are swarmed across a bare-bones physical infrastructure.

Infrastructure as Code toolchain

Depends heavily on the backend infrastructure used; i.e. cloud, bare-metal

In Amazon’s Cloud, their CloudFormation templates allow you to specify instances, network, load balancers, and even security settings for your infrastructure in JSON format. Microsoft Azure has Azure Resource Manager templates that do much the same thing.  Terraform, allow you to specify all these settings in a more abstract fashion, with the idea that it will work across multiple cloud providers.

Unfortunately, truly integrated tools that allow you to model and configure both infrastructure and application at the same time are less common .Ubuntu’s Juju is an open source tool that allows you to model and control the infrastructure and services running on it together. This is common in Docker world.

Configuration Management tools include; Chef, Puppet, Ansible, Salt, CFEngine.

These configuration management tools allow you to specify recipes for how a system should get built. This includes the OS dependencies, system configuration, accounts, SSL certs, and most importantly, your application.

While Chef is more about configuring infrastructure, Habitat extends down into the application build cycle, and then into the application deploy cycle, bookending and complementing Chef.

Service Discovery Tools: Etcd, ZooKeeper and Consul are a few common tools to perform service discovery and state tracking across your infrastructure. They keep key value information and service health metrics and enable more real time orchestration of your systems

Containers generally don’t use Chef or Puppet as their configuration is generally handled with a simple text file called the Docker file. Docker Swarm, Google’s Kubernetes, and Mesos are three popular Docker platforms that do orchestration of containers. They allow you to bring in multiple hosts and run your container workload across all of them. They handle the deployment, the orchestration and scaling.

Continuous Integration

the practice of automatically building and unit testing the entire application frequently. Ideally on every source code check in. Continuous Delivery is the additional practice of deploying every change to a production like environment, and performing automated integration and acceptance testing. Continuous Deployment extends this, to where every change goes through full enough automated testing. That it’s deployed automatically to production

In a continuous integration build system, on each commit, there’s an automatically triggered build that takes the entire code base, builds it, runs all the unit tests, other code validation steps, and packages an artefact at the end along with a build status and a log. The artefact must be tagged with a build number.

There are six practices that are important when doing continuous integration:

  1. All builds should pass the coffee test (builds should not take longer than 5 min)
  2. Commit really small bits (commit smallest amount of code per commit)
  3. Don’t leave the build broken
  4. Use Trunk based development flow
  5. Don’t allow flaky tests
  6. The build should return a status, a log, and an artefact.

Continuous Delivery

it’s the practice of deploying every change to a production-like environment and performing automated integration and acceptance testing along the way. It must use the same artefacts created in the CI process. Good practice is to set permissions so that the CI system can only write in the artefact in the artefact repository and the deployment system can only read.

How artefacts flow through the system;

  1. Code is checked-in a version control system
  2. The commit triggers a build in your CI system
  3. Once the build finishes, the resulting artefacts are published into a central repository
  4. Deployment workflow to deploy the artefacts into a pre-prod environment
  5. Testing happens (automated as much as possible)
  6. Artefact is released
  7. Artefact is deployed to production

CI/CD Toolchain

Below are the six key phases for continuous delivery and the tooling that’s associated with it.

  1. Version Control: where we commit code changes and can view the entire history of code changes. Tool: Git in the form of GitHub or BitBucket
  2. CI Systems: Jenkins being open-source is popular. Other options include GoCD, Bamboo and TeamCity. CI as a Service offered by Travis CI and Circle CI
  3. Build: Very language dependant. Make or Rake for Ruby, or a workflow based approach with Maven.  For front-end code Gulp is popular
  4. Test:
    1. Unit test: Most development languages have Unit Testing built-in.
    2. Code hygiene: Same as above
    3. integration testing tools include Robot, Protractor and Cucumber
    4. Infrastructure testing tools; Kitchen CI for Chef
    5. Performance testing: ApacheBench or Jmeter
    6. Security testing: open-source tools like Galen and Mitten. To run the whole app locally for the above tests you can use tools such as Vagrant, Otto and Docker Compose
  1. Artefact repository: Artefactory or its open-source equivalent Nexus. Or Docker Hub for Docker image
  2. Deployment: Rundeck (workflow driven)