Some of my projects on github use Sauce Labs as a part of the continuous integration test suite that run on Travis-CI every time I push to those projects. Sauce Labs is a service that offers testing across multiple browser using virtual machines and requires a secret key to access these machines.
Since these keys have to be a secret, they are defined as secure environment variables in the travis configuration file. As the documentation in Travis suggests, they are not available during pull requests (if they were, a malicious user could simple send a pull request that echos the value - and travis would automatically run that pull request and reveal the secret).
With this security comes the inconvenience of not being able to automatically run the tests for a pull request. The two options we have in this case are to either manually download the pull request and run is on a local machine with the proper authorization, or merge it and then try running it. The first case is cumbersome and I usually fall back to the second case. There have been pull requests that break the tests and since they are already into master now, reverting them is a pain.
I noticed that this has been happening a good number of times, and I finally decided to write a script for it. The shell script does the following.
- All pull requests should be made to a new branch called incoming-pr. The master branch is holy and should not be broken
- When a pull request is made, basic sanity tests that do not require secure variables is executed.
- If the basic sanity tests pass, the pull request is accepted into the incoming-pr branch
- Travis now builds the incoming-pr branch with the full test suite.
- If the tests on incoming-pr succeed, the branch is merged into master.
- If the tests fail, a backup of the branch in created by a new name.
- In both cases, incoming-pr branch is deleted and created again to be at par with master. This is done to ensure that we are ready for the next pull request.
- In step 7, when I create a new incoming-pr branch, travis tries to build it again, and no matter what the result, the branch is deleted and created again. This leads to an loop and to break this, we run the scripts only when master and incoming-pr have differences.
- I was initially using a temporary github account to perform pushes, but nschonni indicated here that instead of using passwords, we could use github personal tokens.
- Git push always seems to print the updated branch. In this case, we need to use the quiet flag with pushing and redirect the error stream to dev/null so that no output is printed.