Have you ever stepped back and seen the filthy problems that engulf any software product deployed on production? Some of these problems cannot be solved even if the team/company adopts a stringent set of rules or processes. Yeah!! Here are some nasty ones:
- Incase there is one dedicated Operations team for every product/project, the cost for the project goes drastically high.
- Usually Operations Team is always shared across many projects/products to save the cost within an organisation. In this case, inspite of properly planning the calendar for production deployments in each team, there will be some slippages; due to which production deployment dates of some teams will postpone and hence overlap with others. Ultimately the Operations team gets pulled into different deployments at the same time in an effort to satisfy each teams’ deadlines. Finally the end products/customers suffer from not being delivered on time or the quality of the delivered product not being up to the mark.
- In some cases, delivery times are not certain due to Ops Team not available, due to which the end customers need to wait for new releases and patches.
- Detailed testing is required before every new/patch release and hence more testing resources, time and money gets involved. Even after a clean QA, production might not be defect free due to environment differences, wrong configurations, etc.
If you are following the buzz in the software technology operations world, DevOps is a very happening and rising trend. If this is what is running in your mind – “Oh this article is on DevOps, forget it!!” Beware of the big cost you will pay for not understanding and adopting DevOps at your will.
Here you go :
Myths and rumours about DevOps
What is DevOps?
USP of DevOps
Mantra of DevOps
Different parts of DevOps
Why the product owners/teams love DevOps
Why the customers/end-users love DevOps
Future of DevOps
- DevOps is a new vertical/department apart from the existing 3 verticals – Development Vertical, Testing Vertical and Operations Vertical.
- DevOps is a new process invented recently.
- DevOps means developers need to be part of operations and operations team need to be part of development.
- DevOps is a new tool.
- DevOps is meant for projects hosted on cloud or for start-up companies only.
- DevOps suits only huge teams or complex products.
- Its a waste of time and effort to invest in DevOps.
- DevOps eats away lot of time and money, hence better to avoid it.
Is a blend of development and operations in simple terms.
Brings development, testing and operations team under one unified umbrella and preaches that all of them are equally responsible for any changes in the product delivered on production. Sounds philosophical!!? Read more …
Will help produce high-quality software and deliver the same to the customers as quickly as possible.
Multiple production deployments of a product in a single day, is the power of DevOps.
Development wants to make changes – give new patches, features, versions to the existing product(on production). Operations want stability and avoid changes (in some extreme cases reluctant to change the production deployment fearing the stability).
DevOps is the game changer which strikes a good balance between change and stability. DevOps help release new/enhanced products to customer which are stable and of high-quality.
‘automate automate automate’ is the sole mantra.
- Automate infrastructure
- Automate builds
- Automate tests (from unit tests, integration tests, behavior driven tests) to validate if it meets the acceptance criteria
- Automate deployments
- Automate application monitoring post-deployment
- Automate error notification to the appropriate teams
Infrastructure as Code
Setting up Infrastructure has always been a pain point across all teams – Dev, QA, Ops
Till recent past, it had to be setup manually.
Developers used to setup their local env after lot of trial and error method. They had to document the steps so that it can be used by other team members to setup their local env or used by Ops team to setup different envs like QA + Perf, UAT (User Acceptance) and Prod. This is a real cumbersome and complex process. There is a lot of probability that the document may miss some steps or someone following this document may miss out a few steps. Hence this method on the whole is error prone.
Another aspect of this approach is that setting up an env manually eats away lot of time and effort.
Infrastructure as Code is a new approach to combat this problem. Code is written, when run translates into a fully blown infrastructure with the required software installed on it with no manual intervention. These scripts can be run as many number of times required and hence all the envs are identical and faster to setup with no errors.
Use Vagrant to setup virtual machines.
Use Chef, Puppet, Ansible as configuration management tools to provision software on these VMs.
Continuous Integration + Continuous Testing = Continuous Delivery
In any software team, team members do code checkins often, good practice being on a daily basis. All this happens to a central code repository which is shared by the whole team. If one code checkin introduces a bug in functionally of the product, it becomes hard for the developers to go back and see what change or code checkin introduced that bug. This results in a lot of time and effort spent to debug and fix the bug. This will eventually leads to blame game.
Continuous Integration means that the changes to code is getting integrated and tested continuously. For every code checkin, the CI Server should automatically kick off the build process which will run the unit tests and integration tests to see if nothing breaks and hence prevents introduction of bugs. If there is a bug, the build fails indicating a failed test and informs the same to the developer who checked in the code. This will help the respective developer to go back and look at the changed set of files and fix the bug immediately. Also, code coverage tools are integrated to analyse the code and check the quality.
Use Git for source code version control.
Use Jenkins, Hudson, CruiseControl as the CI servers to run automated builds for every code checkin and invoke the tests.
Use JUnit, TestNG for writing unit tests.
Use SonarQube for code coverage and analysis.
In the past, whenever a new build was built there was no way (except manually) to confirm if all the functionalities worked as expected and no new error was introduced. This resulted in lot of man hours being spent to do smoke tests and also some important test flows by the QA team. Even after this, there was no guarantee that the build was defect free.
Now is the age of behaviour driven development. The product is seen as a set of behaviours. Each functionality has an associated behaviour. Acceptance criteria are framed by the BA or the end customer for each behaviour. Based on this, automation testers write tests for every acceptance criteria called acceptance tests. When a new build is invoked, these acceptance tests are executed automatically and only if all these tests pass is the build considered good and ready for production. But care needs to be taken that all the behaviours are covered. Test execution report is generated as part of this.
Use cucumber to define the acceptance criteria and use appropriate framework to automate the tests.
Continuous Delivery means that the integrated code changes which is the latest, is being deployed to a designated environment and a suite of tests are run on it to ensure it is production ready.
In earlier days, promoting the builds manually based on the test results shared by developers/QA used to be the norm. This involved lot of time and effort. Moreover manual tracking of the eligible build numbers can be error prone and add confusion.
With continuous integration and testing, automatically the CI Server identifies which builds are good and eligible to be promoted to the next environment. These builds are stored in artefact repository. Using this, the eligible build moves to QA where QA runs their set of functional and performance tests with different data sets. If it goes fine, the build is moved to the next env like UAT where it undergoes its own set of tests and finally gets moved to Production env (if the UAT owner promotes it to Production).
The end result – as and when code checkins happen by developer, the code is tested and delivered to Production without any fear of failing and with very little manual intervention.
Use Nexus, Artifactory as binary repository manager to store and maintain different versions of binaries.
Log Aggregation and Monitoring
If releasing new builds to Production is the biggest nightmare for Ops Team, supporting the product on Production env post-deployment is the next biggest nightmare. The application might have multiple log files and Ops team might have to setup jobs/custom scripts to monitor each of them and apply all the error rules shared by Dev Team. Also when the error is notified or when customers report an error, a ticket needs to be raised manually and assigned to the L3/L4 support team; wait for the ticket to be completed and then notify the end customers about the solution to the error. Lot of delay due to manual intervention and hence more possibility of slipping the SLA(service level agreement).
Log Aggregation tools when integrated with Production machines gives a huge benefit. If the log files are scattered, log monitoring gets a performance hit. So the log aggregator gives consolidated view of logged data; monitoring on this consolidated log makes it more powerful and faster. Alerting and tracking tools collect information from monitoring tools and open tickets automatically to the appropriate teams; this reduces the turnaround time for production errors and helps meeting SLA.
Use Sumologic, Loggly for log aggregation.
Use Sensu, New Relic for application monitoring.
Use PagerDuty for alerting and tracking.
- Most of the defects/bugs are caught and resolved in non-production environments. The net result is that, very few bugs are encountered in Production environment.
- When defects/bugs are encountered in Production, it takes lesser time to fix, test and release its patch back to production, owing to the automated builds, tests and deployments.
- Due to the stability of the product, the post-deployment maintenance costs for the software in production environment goes down drastically.
- If any production deployment fails, it takes very minimum effort and time to revert back to an earlier working version.
- Ops Team don’t have to depend on the Development Team for the detailed steps to setup the Production environment and also the first aid solutions to support ‘the most frequently encountered error catalogue’.
- Development Team doesn’t have to wait for the Operations Team to set up the Production env and install the new version of the product.
- New Features/Products can be launched quickly and more frequently in the market.
- The products launched are of high-quality and more stable.
- Less fear and more confidence while launching new products/versions.
- High-quality of the product translates into higher revenues and happy customers.
DevOps is being adopted by more and more organizations.
Its turning out to be more of a norm which you cannot avoid wherever you go.
Non-adherence to DevOps will put the company’s future at peril; by delivering an unstable product and customers not forgiving this humongous sin.
! ! ! Statutory Warning ! ! !
DevOps is a terrific drug! One sniff and you can never quit!
DevOps takes all the burden of product delivery on its shoulders, relieves you from tensions of production deployment and support; let’s you lead a hassle free life.
Beware of the extra pounds of fat and free time you gain from this life style!