The following is a guest post by Florian Motlik. Florian is CTO of Codeship, a hosted Continuous Integration and Continuous Deployment platform. What the heck is that, you might ask? It’s kind of a philosophy and toolset for working on websites. That is perhaps a bit too simplified, so I’ll let Florian explain.
A great development workflow will make your good developers great, and your great developers exceptional. A bad workflow will take your best engineers productivity down to a junior level and make wish he worked elsewhere.
It is critically important for every team and especially the leaders of those teams to get their development workflow in order. Two best practices that have gained a lot of traction over the last few years are Continuous Integration and Continuous Deployment. Here’s some quick definitions.
Continuous Integration is the practice of testing each change done to your codebase automatically and as early as possible.
Continuous Deployment follows the testing that happens during Continuous Integration and pushes changes to a staging or production system. This makes sure a version of your code is accessible at all times.
First, we will take a close look at Continuous Integration and show you how to get started with testing. Then we will move on to Continuous Deployment which is the next logical step.
With both in place, your development team will be orders of magnitude more efficient.
Automation is a cornerstone of a great development workflow. Every task that can be done by a machine should be. Automation gives you the time to focus. Testing is one such task.
Through testing, you can be sure that the most important steps your customers will take through your system are working, regardless of the changes you make. This gives you the confidence to experiment, implement new features, and ship updates quickly.
Start testing at the top
Good testing begins with viewing the app from a user’s perspective. What are the typical workflows? What features and steps do users go through often?
Then start testing with those in mind. Use tools like Selenium or PhantomJS to make sure those important workflows work.
A great way to determine the most critical workflows is by getting everyone in your team to write down a list of them and then combine those lists. You can read an in-depth guide to this technique in one of our blog posts.
Here is a list of tools that you can use to start testing UI:
- Selenium (Frontend Testing)
- CasperJS (Frontend Testing)
- Cucumber or Lettuce (Behaviour Driven Testing for Ruby and Python)
From this strong base of tests that make sure the workflows are working, you can go deeper and start unit testing different parts of your site. But beware: Your customers want your UI to work, so make sure it does with every change!
With all those tools in hand, the single most important step you can take for testing is getting started. It pays off immediately, and not just from a tech perspective. You can read about the business advantages of Continuous Integration in this blog post by Joe Green.
Run your tests continuously
Humans make mistakes. Sometimes people consider a change so small it can’t possibly break the app, so they don’t run tests. Then your product goes down and nobody knows why.
Automated tests can only show their true power when you run them continuously and for every change. They have to be run on an automated and separate system to prevent the ‘It works on my machine’ syndrome. Only if the Continuous Integration infrastructure tells you the tests pass, the tests pass.
The advantage of running all tests immediately for every change is that you know right away if something broke. Consider working on a 2 week sprint. On day two of the sprint, one of your developers changes something that breaks a test. If you don’t continually run all of your tests, there is a chance nobody detects the problem. People keep working, and at the end of the sprint, you run the whole test suite again. The test fails. It now has been 2 weeks since the breaking change was committed. Finding the change that broke the tests is incredibly hard, as there now have been a lot of commits since then.
Lots of teams are on way longer cycles than 2 weeks. This is incredibly wasteful and can be easily prevented by running all your tests every time one of your developers pushes code to the repository. Any failures can be fixed right away.
Continuous Integration is the first, but very important step in Continuous Deployment.
If you’d like to learn more about it, Martin Fowler has released a great and very in-depth article about Continuous Integration on his blog
Code rots if it doesn’t run somewhere. External dependencies are updated, server packages installed, API’s change or operating systems replaced. Often we do not have the ability to control every part of our infrastructure. We rely on other products and companies to maintain parts of our stack. All of these changes can break your app without you realizing it.
This is why Continuous Deployment is incredibly important. Whenever the main development branch in your repository passes all the tests, you should push your code at least into a staging environment or in the best case into production. Your development or QA team can then review the latest version of your application and give feedback. Furthermore you can run tests against that staging system to make sure it works in a production-like environment.
This helps you find problems early and fix them while it is still cheap. An important concept to understand when starting with Continuous Deployment are deployment pipelines.
A deployment pipeline defines all steps necessary to go from your codebase to the production website. Automating these steps is crucial to have fast deployments. But often, deployment pipelines also include manual steps, like a product manager signing off on a new release.
Deployment pipelines have to be easy to follow and replicate. This allows you to push new changes regularly. Especially when you need to ship a fix, predefined steps can save you from bringing more harm to your site.
Once you set up a Deployment Pipeline and automate the steps, you are in a position to ship small changes quickly, get feedback from your customers, and iterate on your product. This is a major competitive advantage to other companies who need a long time to react to customer feedback. Intercom did a great blog post about this.
It’s amazing how much lower the cost of bringing a new software product to market has gotten in last couple of years. It has never been easier or cheaper to build great software. This levels the playing field between you and your competition. You need to ship new features faster than your competition while delivering better and stable long term quality. Continuous Deployment is one of the techniques that will help you to get there.
How can I get started?
After interviews with dozens of companies and the feedback from thousands of developers using our product we’ve identified several steps you can take to get started with Continuous Deployment.
Test test test, but be smart about it
Testing is the first step that needs to be taken. You need to get to a point where you can change almost any part of your system and be sure you didn’t break any other feature because your tests will catch those errors. Once you are at this point, it liberates you from having to think about impacts this change might have. You can now experiment with your product and iterate to build exactly what your customers want and pay for.
Tests are not there to make your developers happy. They are there to build a stable and high quality product for your users. They are a tool to keep your users happy and introduce changes without destroying their experience. Building a product without properly and automatically testing (as your users would use it) means working with the hands tied behind your back. You might get new features out there, but you will never be as fast and good as you could be.
Testing is the basis on which Continuous Deployment is built. Tests make software as Clemens, one of our developers, explained in his blog post.
This script tests Google Translate to make sure it translates “good morning” from German to English:
#Run with casperjs test translate.coffee casper.start 'http://translate.google.com/', -> @.sendKeys('#source', 'Guten Tag'); @.then -> @.click '#gt-submit' @.waitForText('Good day') @.then -> @.test.assertSelectorHasText '#result_box', 'Good day' @.then -> @.capture 'output.png' casper.run -> @.test.done(1)
CasperJS helps you to start testing your application quickly, and it will scale up to test more complex scenarios as you need.
Every step in your deployment should be automated. Once set up, you should never have to think about which steps need to be taken. Every choice and every manual task leads to possible mistakes, so you simply do it less often. By automating every part of your deployment you can push new features, but more importantly, push fixes very quickly. Continuous Deployment works because of automation, so don’t skip this step.
When pushing quickly you also want to be able to go back to an earlier state easily. Otherwise you won’t feel safe enough to do as regularly as you should be.
Automated Rollback includes your codebase as well as your database. Make sure you regularly backup your database and use those backups to recreate your system on test machines. If you don’t regularly restore your backups you can’t be sure they will work when you desperately need them.
Deploy to Staging
Create a staging environment and push your master branch or main line of development into that Staging application any time changes are made. This Staging application is your first line of defense and helps to review all changes before going into production. You can even run automated tests against Staging to make sure it works as expected.
Use your staging environment
If you use your product in your daily work, use your Staging and not your Production server. As you push changes continuously to Staging, you will see first hand when something is broken as it immediately impacts your team. “Eating your own dog food” makes all of your developers aware when the quality drops.
Automatically deploy to Production
As a final step, continuously deploy to your Production environment. This will truly change the way your organization works. There will be no more fear in changing your site. This means you can iterate quickly on new features, but if you ship a bug, you can quickly ship a fix as well. You are not tied to arbitrary cycles that slow down your team.
Moving to Continuous Deployment will change your development dramatically. It will make you more productive and lead to a more stable and better product. There are steps like automated testing that you need to get started with first. Automated testing is not a tool to build better software, but a process to build a better product for your users. Once set up, you can then reap the benefits of deploying new versions several times a day.
Please let us know if any questions come up or if there is anything else we can help with either here in the comments or send us a tweet to @codeship . We’ve also recently released a free eBook that walks you through our development workflows and dives deeper into Continuous Deployment. You can grab the book here.
Ship long and prosper!
I can’t stress enough how important continuous integration is.
Healthcare.gov didn’t have it… Google, Amazon, have set the gold standard.
I have been fortunate to work the last 13 months for two companies doing this real well and it’s amazing how efficient and agile we are. It’s also amazing the how the majority of our web developers embrace dev ops and have grown in that area as well.
Yes CI definitely is one of the most important tools for any team. It just forces you into a great workflow. Have you guys tried deploying after the CI process as well? Any experience in your project there?
Good advice! I’ve never worked in a team, but after every change I reload the page and see if the new edit broke anything or not.
While that is ok as a workflow automating that process helps a lot so you know that your page works even though you might not take a look at every page.
I have seen this kind of approach becoming more and more the common case, but there is no way I could agree with all, and I can’t understand how developers are not raising hands agains this:
You should never go to Production without QA, release an stable and solid version, you should aviod the go to production without testing (1, QA, UAT)… and we would you need to push changes so regularly?, if your code and application is stable it should require a minimum amount of online-onthefly updates (2 must not happen)
Whenever I see this kind of parading I feel like developers are becoming lazy and irresponsibly, you must avoid issues… updates should contain a minimum amount of bug fix and only new features… but modern developers are using the final client as they QA, and what is worst is the the final users are allowing it, they want to see weekly updates on they applications… why should we test in Production? what is QA doing? why we teach our users to test and accept our bug and bad quality software?
(Please notice that I only disaggregated the go to production and lest just fix if broken, we should warranty that our apps are bug free)
We’ve found the exact opposite to what you mentioned. By pushing regularly we are forced to put a lot more effort into our tests and automating testing. It pushes you to have every part of your application well tested.
QA should always be part of the development process. Handing of code to a seperate team which have to deal with any problems doesn’t put the right quality emphasis on the development team.
In general it forces us to build better automated quality controls while it allows us to iterate quickly on our product and build something for our customers.
Does this seem like a good foundational goal?
yes. You don’t need these fancy tools to practice CI in a small team, honestly even some larger teams too.
In large teams, it’s probably best to standardize on some explicit CI technology, but I’m not sure how large the team needs to be to make that necessary.
Some of the tools mentioned here are complex and powerful, and I think you probably need to have a large team to “waste time” (as it were) learning to use them effectively, beyond the hello world examples here.
If you are small, keep it simple.
setting up the first tests and the whole automated workflow for CI is a great first step. We typically tell new users to just write one test (which can be as easy as just calling the site or logging in) and building the whole automated workflow with this one test. As soon as you have everything set up that it just runs automatically all the time you get into a “well why not just write a test for it” mode, as the whole CI is already set up any way. Teams we talked to expanded from this first step very quickly to testing everything and really getting into TDD.
True. Check github page of my project (refactor branch is fresher) for an example of a Grunt-based CI development process:
coffee/md/less files are compiled to js/html/css on file save
tests are executed on each page reload.
You just type
grunt, edit files, reload browser (haven’t bothered with auto-reload yet :)).
I’m going to evaluate Brunch.io instead of Grunt, seems promising.
We are using CI in our latest project…it would be nice if we would have time to fix our unit tests (There are currently about 5% broken).
How ever it is really helpful for the most parts as you can see immediately when a part of the application breaks. So I have to say, I love it and would never consider to work without it for a bigger project.
I’ve talked to lots of teams that had the same problem with a couple of tests always failing, the problem there is that you easily get into a workflow of just ignoring the tests in general. Had that in earlier companies I worked in as well and it always was worthwile to either fix those tests or remove them if they aren’t necessary anyway. It just lowers the usefulness of all tests if even only one of them consistently fails
Selenium was a lifesaver for testing a donation script we had put together. We actually used the firefox extension Selenium IDE and recorded the typical steps a customer would use to make sure each step was working along the way. One of the form buttons broke at one point from a small code change, so daily running picked it up. It was fairly easy to set up, although the plugin has to be started manually, but once you click start it does all the work for you.
We use Selenium for all of our Frontend Testing and we depend on it to find problems every single day. Being able to confidently say that the most important features of your app work reliably for your users is very helpful