From 36c76a62b8e3515d3ede8fb4c2cb67a4b19837e5 Mon Sep 17 00:00:00 2001 From: "Jay R. Newlin (PromptWorks)" Date: Wed, 27 Nov 2019 16:08:42 -0500 Subject: [PATCH 1/6] Remove Ghost Inspector details from base README --- README.md | 29 ++--------------------------- 1 file changed, 2 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index d18177d4..44749862 100644 --- a/README.md +++ b/README.md @@ -270,33 +270,8 @@ execute UI tests than vanilla Selenium. Ghost Inspector tests and steps can be exported to files that the Selenium IDE can import. We export these tests/steps regularly and archive them with the AT-AT codebase in the `uitests` directory. -To run the Ghost Inspector tests against a local instance of AT-AT, -you will need the following: - -- [docker](https://docs.docker.com/v17.12/install/) -- [circleci CLI tool](https://circleci.com/docs/2.0/local-cli/#installation) -- the prerequisite variable information listed [here](https://ghostinspector.com/docs/integration/circle-ci/) - -The version of our CircleCI config (2.1) is incompatible with the -`circleci` tool. First run: - -``` -circleci config process .circleci/config.yml > local-ci.yml -``` - -Then run the job: - -``` -circleci local execute -e GI_SUITE= -e GI_API_KEY= -e NGROK_TOKEN= --job integration-tests -c local-ci.yml -``` - -If the job fails and you want to re-run it, you may receive errors -about running docker containers or the network already existing. -Some version of the following should reset your local docker state: - -``` -docker container stop redis postgres test-atat; docker container rm redis postgres test-atat ; docker network rm atat -``` +For further information about Ghost Inspector and its use in AT-AT, check out [its README](./uitests/README.md) +in the `uitests` directory. ## Notes From a016cfae289731a74e486c53d942989701abcfdc Mon Sep 17 00:00:00 2001 From: "Jay R. Newlin (PromptWorks)" Date: Wed, 27 Nov 2019 16:09:24 -0500 Subject: [PATCH 2/6] Start Ghost Inspector README with details extracted from main README --- uitests/README.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 uitests/README.md diff --git a/uitests/README.md b/uitests/README.md new file mode 100644 index 00000000..18d92351 --- /dev/null +++ b/uitests/README.md @@ -0,0 +1,29 @@ +# Ghost Inspector Readme + +To run the Ghost Inspector tests against a local instance of AT-AT, +you will need the following: + +- [docker](https://docs.docker.com/v17.12/install/) +- [circleci CLI tool](https://circleci.com/docs/2.0/local-cli/#installation) +- the prerequisite variable information listed [here](https://ghostinspector.com/docs/integration/circle-ci/) + +The version of our CircleCI config (2.1) is incompatible with the +`circleci` tool. First run: + +``` +circleci config process .circleci/config.yml > local-ci.yml +``` + +Then run the job: + +``` +circleci local execute -e GI_SUITE= -e GI_API_KEY= -e NGROK_TOKEN= --job integration-tests -c local-ci.yml +``` + +If the job fails and you want to re-run it, you may receive errors +about running docker containers or the network already existing. +Some version of the following should reset your local docker state: + +``` +docker container stop redis postgres test-atat; docker container rm redis postgres test-atat ; docker network rm atat +``` From 75aa82f9386df4273546912b0f107e817f6c833b Mon Sep 17 00:00:00 2001 From: "Jay R. Newlin (PromptWorks)" Date: Wed, 27 Nov 2019 16:30:02 -0500 Subject: [PATCH 3/6] Added introductory information --- uitests/README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/uitests/README.md b/uitests/README.md index 18d92351..986cc1d6 100644 --- a/uitests/README.md +++ b/uitests/README.md @@ -1,5 +1,12 @@ # Ghost Inspector Readme +The suite of Ghost Inspector tests that runs as part of the CI/CD workflow +can be found [here](https://app.ghostinspector.com/suites/5d9e3d303ab5d56633c11598). Its current status is https://api.ghostinspector.com/v1/suites/5d9e3d303ab5d56633c11598/status-badge + +[Another suite](https://app.ghostinspector.com/suites/5d9603f1af31210914da04ca) of Ghost Inspector tests runs once daily against the Staging site. Status: https://api.ghostinspector.com/v1/suites/5d9603f1af31210914da04ca/status-badge + +## Running Ghost Inspector tests locally + To run the Ghost Inspector tests against a local instance of AT-AT, you will need the following: From 83d90007609d98ab345a3b2a310c05a8c1d7a0d2 Mon Sep 17 00:00:00 2001 From: "Jay R. Newlin (PromptWorks)" Date: Wed, 27 Nov 2019 16:40:38 -0500 Subject: [PATCH 4/6] Add Testing Philosophy section --- uitests/README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/uitests/README.md b/uitests/README.md index 986cc1d6..1ab5777a 100644 --- a/uitests/README.md +++ b/uitests/README.md @@ -5,6 +5,18 @@ can be found [here](https://app.ghostinspector.com/suites/5d9e3d303ab5d56633c115 [Another suite](https://app.ghostinspector.com/suites/5d9603f1af31210914da04ca) of Ghost Inspector tests runs once daily against the Staging site. Status: https://api.ghostinspector.com/v1/suites/5d9603f1af31210914da04ca/status-badge +## Testing philosophy + +The tests have been created to traverse the most common user flows in AT-AT. There are a few tests (e.g. "New Portfolio - no optional fields") +that check for regressions. Others (e.g. "Remove Portfolio Member") check less-common, "negative path" flows. Tests are added as necessary +to ensure fairly thorough checking of AT-AT. + +The tests are constructed in a "stepwise" fashion; that is, no individual test depends upon another, and each test checks one step in complex +user flows. As an example, there are six tests for creating a new Task Order, one corresponding to each screen in the process plus one +for adding a second CLIN to a TO. These tests rely on Ghost Inspector's "Import steps from Test X" functionality to perform all the +necessary setup for the current test. This also ensures that tests can be run in any sequence because Ghost Inspector launches up to four +tests simultaneously. + ## Running Ghost Inspector tests locally To run the Ghost Inspector tests against a local instance of AT-AT, From e7cdcd349712a7ca757616a153a56edcae4b2b30 Mon Sep 17 00:00:00 2001 From: "Jay R. Newlin (PromptWorks)" Date: Wed, 27 Nov 2019 16:58:21 -0500 Subject: [PATCH 5/6] Add section on dealing with changes to the UI and failing tests --- uitests/README.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/uitests/README.md b/uitests/README.md index 1ab5777a..5a119302 100644 --- a/uitests/README.md +++ b/uitests/README.md @@ -1,6 +1,6 @@ # Ghost Inspector Readme -The suite of Ghost Inspector tests that runs as part of the CI/CD workflow +The suite of Ghost Inspector tests that runs as part of the CI workflow can be found [here](https://app.ghostinspector.com/suites/5d9e3d303ab5d56633c11598). Its current status is https://api.ghostinspector.com/v1/suites/5d9e3d303ab5d56633c11598/status-badge [Another suite](https://app.ghostinspector.com/suites/5d9603f1af31210914da04ca) of Ghost Inspector tests runs once daily against the Staging site. Status: https://api.ghostinspector.com/v1/suites/5d9603f1af31210914da04ca/status-badge @@ -17,6 +17,21 @@ for adding a second CLIN to a TO. These tests rely on Ghost Inspector's "Import necessary setup for the current test. This also ensures that tests can be run in any sequence because Ghost Inspector launches up to four tests simultaneously. +## Handling UI changes + +As with any UI-testing system, Ghost Inspector tests will fail because of changes to the UI. This can be problematic since failures +cause errors in the CI workflow, which could cause the Pull Request not to be deployed. To mitigate this issue, we have utilized +the following strategies: + +1. Front-end developers notify the project QA lead of changes that will potentially cause Ghost Inspector failures --OR-- the +QA lead monitors all failures on the CI suite and determines that such a change has occurred. + +2. If only one step in a test is affected (e.g. a visual but not functional change), that step can be marked as "optional" in +the Ghost Inspector UI. + +3. If the (potentially) failing test will need to be reworked to account for functional changes, the test is moved from the CI +suite to the "Holding" suite until the PR is merged. Then the test can be edited and returned to the CI suite. + ## Running Ghost Inspector tests locally To run the Ghost Inspector tests against a local instance of AT-AT, From c23d01bc3dc1cce4fb44e827d238f5cfd70841be Mon Sep 17 00:00:00 2001 From: "Jay R. Newlin (PromptWorks)" Date: Wed, 27 Nov 2019 17:06:49 -0500 Subject: [PATCH 6/6] Minor edits --- uitests/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/uitests/README.md b/uitests/README.md index 5a119302..0f3a940b 100644 --- a/uitests/README.md +++ b/uitests/README.md @@ -13,9 +13,9 @@ to ensure fairly thorough checking of AT-AT. The tests are constructed in a "stepwise" fashion; that is, no individual test depends upon another, and each test checks one step in complex user flows. As an example, there are six tests for creating a new Task Order, one corresponding to each screen in the process plus one -for adding a second CLIN to a TO. These tests rely on Ghost Inspector's "Import steps from Test X" functionality to perform all the -necessary setup for the current test. This also ensures that tests can be run in any sequence because Ghost Inspector launches up to four -tests simultaneously. +for adding a second CLIN to a TO. Each test relies on Ghost Inspector's "Import steps from Test X" functionality to perform all the +necessary setup before running its own check. This also ensures that tests can be run in any sequence since Ghost Inspector launches up to +four tests simultaneously. ## Handling UI changes