Litmus tests are (non-automated) tests that are documented in the litmus database. See http://litmus.mozilla.org.
If you're interested in automated testing, see Mozilla automated testing.
Smoke Tests
There are types of litmus tests. Smoke tests were the first developed. These were developed before litmus, but then pulled in. A "smoke test" is a quick check, done to verify the basic functionality of a certain feature, or a sanity check of the build process.
BFTs, or Basic Functionality Tests
This set of tests would take longer to run, but will cover an entire release. It may not cover it at much depth, but it does try to cover the major features.
FFTs, or Full Functionality Tests
This is supposed to be more comprehensive and will take longer to run.