- Where can I see the build system status?
- What builds are displayed on the build system status page?
- tamarin-redux mercurial builds are displayed in the top 3 sections: Compile, Smoke, Test
- sandbox builds are displayed in next 3 sections: Sandbox Compile, Sandbox Smoke, Sandbox Test
- tamarin-central is considered a "sandbox" and will appear under Sandbox phase
- What are the build phases?
- Compile, Smoke, Test. When all of the slave machines are free they all start the first build phase simultaneously. If an error or failure occurs on any slave in a phase the build stops. The next phase is started when all slaves have completed the previous phase successfully.
- The Compile phase compiles all of the tamarin source code and builds all of the shell executables, any errors will stop the phase and a red box will appear on the slave where the error occurred. The compile phase takes about 20-25 minutes.
- The Smoke phase runs a set of tests on each platform until 2 minute is exceeded. The set of smoke tests can contain any shell build and any test suite (including performance). The smoke test list is constantly updated to contain the most recent failures, most frequent failures, and the newest testcases. The idea is to catch test failures in a few minutes rather than running the complete test run (1h 45m). The smoke test including downloading the shell takes about 3-4 minutes.
- The Test phase runs all functional testcases including acceptance tests, cmdline tests, and selftests. The acceptance tests are the majority and are run on all shells (release, release-debugger, debug, debug-debugger). The test phase takes approximately 1 hour 45 minutes.
- The Performance and Deep phases do not run synchronously. The phases run when slaves are available and the other phases do not wait for completion to start the next build.
- How are builds triggered?
- checkins (hg push) to http://hg.mozilla.org/tamarin-redux and sandbox requests will trigger a buildbot build
- When all build slaves are idle the next build begins
- Before the build begins buildbot waits for 2 minutes of no checkin activity
- The build system prioritizes tamarin-redux checkins over sandbox builds
- When several tamarin-redux checkins occur the newest change is built including all checkins since the last build
- Each sandbox build occurs in the order they were submitted, The submitter may cancel a build.
- How do I build a sandbox build?
- How do I view and change the queue?
- If buildbot is busy the queue is displayed http://tamarin-builds.mozilla.org/build_trigger/requestbuild.cfm
- The submitter of a sandbox build may remove a build request by clicking on the Cancel button
- tamarin-redux builds are higher priority than sandbox builds, they cannot be removed but the most checkin including all new checkins are built
- How can I run buildbot scripts locally?
- The buildbot scripts are located in the tamarin repository, get a current copy in your test repository
- cd into the build/buildbot/slaves directory: cd build/buildbot/slaves
- copy the your platform scripts directory into the scripts directory and cd into the new scripts directory: cp -r mac-intel-10_5/scripts . ; cd scripts
- edit environment.sh, change the basedir and buildsdir settings
- (around line 51) basedir=~/hg/tamarin-redux (path to my test repository)
- (next line) buildsdir=~/hg/builds (a directory to store downloaded builds)
- always set current working directory to the scripts directory when running a script
- run a script (e.g.) ../all/run-acceptance-release.sh <optional hg revision number like 1902>
- How do I navigate the build status page?
-
- The page is divided into the phase table at the top and the waterfall table below
- The phase table shows all phases at the top of the page, each slave is shown as a box with a red,green, or yellow color: red means failed, green passed, and yellow has warnings.
- The phase state is idle or active
- Click on the phase (e.g. Compile, Smoke, or Test to show the build slaves from the most recent phase run
- or Click on an individual box to display the test result
- Below the phase table the waterfall displays each buildstep and result
- the left side contains the time column and the user/revision link to represent the revision triggering a build, mouse over the user to see the revision and submission notes
- Each buildstep run on a slave is a box with color indicating status, red is failed, green passed, yellow has warnings
- click the stdio link to see the output
- click on the Build yellow box at the bottom of a set of buildsteps to see the result of all of the buildsteps for a build and the time each buildstep took
- scroll down to see previous builds triggered
- at the bottom of the page click on the [next page] link to display more previous builds
- What are the goals of the build system (QOS)?
- to provide quick and accurate results of building and testing tamarin on supported platforms and configurations
- to represent the build results in a clear and concise manor
- Please give use feedback if you have any comments or suggestions to improve the build system
- How to add tests to the smokes?
- The list of smoke tests are located in the tamarin-repository, test/run-smokes.txt, assume the start directory is test format is cd testdir; command to run test
- above the test should be a # comment describing why the test in the smokes, when test failed, possibly a bugzilla bug
- The tests can be run by following the above instructions for running buildbot scripts locally and executing the all/run-smoke-tests.sh
- As a rule any test failure should be immediately added to the top of the smoke test list so the list is prioritized
- How to exclude tests from acceptance or mark them as known failures?
- The test/acceptance/testconfig.txt file contains a list of tests marked as expected failures or skipped with the configuration
- see the testconfig.txt for current instructions
- generally the format is comma separated list with regular expressions for test and configuration. expectedfail or skip
- expectedfail means the test is run and the assertion is the test fails, if the test passes and is marked expectedfail the test result is unexpectedpass and treated like a failure
- skip means the test is not run, the test may fail intermittently or harm the system altering the results of other tests
