diff --git a/README.md b/README.md index d18177d4..44749862 100644 --- a/README.md +++ b/README.md @@ -270,33 +270,8 @@ execute UI tests than vanilla Selenium. Ghost Inspector tests and steps can be exported to files that the Selenium IDE can import. We export these tests/steps regularly and archive them with the AT-AT codebase in the `uitests` directory. -To run the Ghost Inspector tests against a local instance of AT-AT, -you will need the following: - -- [docker](https://docs.docker.com/v17.12/install/) -- [circleci CLI tool](https://circleci.com/docs/2.0/local-cli/#installation) -- the prerequisite variable information listed [here](https://ghostinspector.com/docs/integration/circle-ci/) - -The version of our CircleCI config (2.1) is incompatible with the -`circleci` tool. First run: - -``` -circleci config process .circleci/config.yml > local-ci.yml -``` - -Then run the job: - -``` -circleci local execute -e GI_SUITE= -e GI_API_KEY= -e NGROK_TOKEN= --job integration-tests -c local-ci.yml -``` - -If the job fails and you want to re-run it, you may receive errors -about running docker containers or the network already existing. -Some version of the following should reset your local docker state: - -``` -docker container stop redis postgres test-atat; docker container rm redis postgres test-atat ; docker network rm atat -``` +For further information about Ghost Inspector and its use in AT-AT, check out [its README](./uitests/README.md) +in the `uitests` directory. ## Notes diff --git a/uitests/README.md b/uitests/README.md new file mode 100644 index 00000000..0f3a940b --- /dev/null +++ b/uitests/README.md @@ -0,0 +1,63 @@ +# Ghost Inspector Readme + +The suite of Ghost Inspector tests that runs as part of the CI workflow +can be found [here](https://app.ghostinspector.com/suites/5d9e3d303ab5d56633c11598). Its current status is https://api.ghostinspector.com/v1/suites/5d9e3d303ab5d56633c11598/status-badge + +[Another suite](https://app.ghostinspector.com/suites/5d9603f1af31210914da04ca) of Ghost Inspector tests runs once daily against the Staging site. Status: https://api.ghostinspector.com/v1/suites/5d9603f1af31210914da04ca/status-badge + +## Testing philosophy + +The tests have been created to traverse the most common user flows in AT-AT. There are a few tests (e.g. "New Portfolio - no optional fields") +that check for regressions. Others (e.g. "Remove Portfolio Member") check less-common, "negative path" flows. Tests are added as necessary +to ensure fairly thorough checking of AT-AT. + +The tests are constructed in a "stepwise" fashion; that is, no individual test depends upon another, and each test checks one step in complex +user flows. As an example, there are six tests for creating a new Task Order, one corresponding to each screen in the process plus one +for adding a second CLIN to a TO. Each test relies on Ghost Inspector's "Import steps from Test X" functionality to perform all the +necessary setup before running its own check. This also ensures that tests can be run in any sequence since Ghost Inspector launches up to +four tests simultaneously. + +## Handling UI changes + +As with any UI-testing system, Ghost Inspector tests will fail because of changes to the UI. This can be problematic since failures +cause errors in the CI workflow, which could cause the Pull Request not to be deployed. To mitigate this issue, we have utilized +the following strategies: + +1. Front-end developers notify the project QA lead of changes that will potentially cause Ghost Inspector failures --OR-- the +QA lead monitors all failures on the CI suite and determines that such a change has occurred. + +2. If only one step in a test is affected (e.g. a visual but not functional change), that step can be marked as "optional" in +the Ghost Inspector UI. + +3. If the (potentially) failing test will need to be reworked to account for functional changes, the test is moved from the CI +suite to the "Holding" suite until the PR is merged. Then the test can be edited and returned to the CI suite. + +## Running Ghost Inspector tests locally + +To run the Ghost Inspector tests against a local instance of AT-AT, +you will need the following: + +- [docker](https://docs.docker.com/v17.12/install/) +- [circleci CLI tool](https://circleci.com/docs/2.0/local-cli/#installation) +- the prerequisite variable information listed [here](https://ghostinspector.com/docs/integration/circle-ci/) + +The version of our CircleCI config (2.1) is incompatible with the +`circleci` tool. First run: + +``` +circleci config process .circleci/config.yml > local-ci.yml +``` + +Then run the job: + +``` +circleci local execute -e GI_SUITE= -e GI_API_KEY= -e NGROK_TOKEN= --job integration-tests -c local-ci.yml +``` + +If the job fails and you want to re-run it, you may receive errors +about running docker containers or the network already existing. +Some version of the following should reset your local docker state: + +``` +docker container stop redis postgres test-atat; docker container rm redis postgres test-atat ; docker network rm atat +```