diff --git a/e2e/README.md b/e2e/README.md index 70d3462982..e34124c197 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -139,16 +139,18 @@ These tests are expected to become blocking and gating with assertions as we ext Our file structure follows the type of type of testing being excercised at the e2e layer and files containing test suites which matcher application behavior or our `src` and `example` layout. This area is not well refined as we figure out what works best for closed source and downstream projects. This may change altogether if we move `e2e` to it's own npm package. -- `./helper` - contains helper functions or scripts which are leveraged directly within the testsuites. i.e. non-default plugin scripts injected into DOM -- `./test-data` - contains test data which is leveraged or generated in the functional, performance, or visual test suites. i.e. localStorage data -- `./tests/functional` - the bulk of the tests are contained within this folder to verify the functionality of open mct -- `./tests/functional/example/` - tests which specifically verify the example plugins -- `./tests/functional/plugins/` - tests which loosely test each plugin. This folder is the most likely to change. Note: some @snapshot tests are still contained within this structure -- `./tests/framework/` - tests which verify that our testframework functionality and assumptions will continue to work based on further refactoring or playwright version changes -- `./tests/performance/` - performance tests -- `./tests/visual/` - Visual tests -- `./appActions.js` - Contains common fixtures which can be leveraged by testcase authors to quickly move through the application when writing new tests. -- `./baseFixture.js` - Contains base fixtures which only extend default `@playwright/test` functionality. The goal is to remove these fixtures as native Playwright APIs improve. +|File Path|Description| +|:-:|-| +|`./helper` | Contains helper functions or scripts which are leveraged directly within the test suites (e.g.: non-default plugin scripts injected into the DOM)| +|`./test-data` | Contains test data which is leveraged or generated in the functional, performance, or visual test suites (e.g.: localStorage data).| +|`./tests/functional` | The bulk of the tests are contained within this folder to verify the functionality of Open MCT.| +|`./tests/functional/example/` | Tests which specifically verify the example plugins (e.g.: Sine Wave Generator).| +|`./tests/functional/plugins/` | Tests which loosely test each plugin. This folder is the most likely to change. Note: some `@snapshot` tests are still contained within this structure.| +|`./tests/framework/` | Tests which verify that our testing framework's functionality and assumptions will continue to work based on further refactoring or Playwright version changes (e.g.: verifying custom fixtures and appActions).| +|`./tests/performance/` | Performance tests.| +|`./tests/visual/` | Visual tests.| +|`./appActions.js` | Contains common methods which can be leveraged by test case authors to quickly move through the application when writing new tests.| +|`./baseFixture.js` | Contains base fixtures which only extend default `@playwright/test` functionality. The expectation is that these fixtures will be removed as the native Playwright API improves| Our functional tests end in `*.e2e.spec.js`, visual tests in `*.visual.spec.js` and performance tests in `*.perf.spec.js`. @@ -158,10 +160,12 @@ Where possible, we try to run Open MCT without modification or configuration cha Open MCT is leveraging the [config file](https://playwright.dev/docs/test-configuration) pattern to describe the capabilities of Open MCT e2e _where_ it's run -- `./playwright-ci.config.js` - Used when running in CI or to debug CI issues locally -- `./playwright-local.config.js` - Used when running locally -- `./playwright-performance.config.js` - Used when running performance tests in CI or locally -- `./playwright-visual.config.js` - Used to run the visual tests in CI or locally +|Config File|Description| +|:-:|-| +|`./playwright-ci.config.js` | Used when running in CI or to debug CI issues locally| +|`./playwright-local.config.js` | Used when running locally| +|`./playwright-performance.config.js` | Used when running performance tests in CI or locally| +|`./playwright-visual.config.js` | Used to run the visual tests in CI or locally| #### Test Tags @@ -169,13 +173,15 @@ Test tags are a great way of organizing tests outside of a file structure. To le Current list of test tags: -- `@ipad` - Test case or test suite is compatible with Playwright's iPad support and Open MCT's read-only mobile view (i.e. no Create button). -- `@gds` - Denotes a GDS Test Case used in the VIPER Mission. -- `@addInit` - Initializes the browser with an injected and artificial state. Useful for loading non-default plugins. Likely will not work outside of `npm start`. -- `@localStorage` - Captures or generates session storage to manipulate browser state. Useful for excluding in tests which require a persistent backend (i.e. CouchDB). -- `@snapshot` - Uses Playwright's snapshot functionality to record a copy of the DOM for direct comparison. Must be run inside of the playwright container. -- `@unstable` - A new test or test which is known to be flaky. -- `@2p` - Indicates that multiple users are involved, or multiple tabs/pages are used. Useful for testing multi-user interactivity. +|Test Tag|Description| +|:-:|-| +|`@ipad` | Test case or test suite is compatible with Playwright's iPad support and Open MCT's read-only mobile view (i.e. no create button).| +|`@gds` | Denotes a GDS Test Case used in the VIPER Mission.| +|`@addInit` | Initializes the browser with an injected and artificial state. Useful for loading non-default plugins. Likely will not work outside of `npm start`.| +|`@localStorage` | Captures or generates session storage to manipulate browser state. Useful for excluding in tests which require a persistent backend (i.e. CouchDB).| +|`@snapshot` | Uses Playwright's snapshot functionality to record a copy of the DOM for direct comparison. Must be run inside of the playwright container.| +|`@unstable` | A new test or test which is known to be flaky.| +|`@2p` | Indicates that multiple users are involved, or multiple tabs/pages are used. Useful for testing multi-user interactivity.| ### Continuous Integration @@ -232,7 +238,8 @@ At the same time, we don't want to waste CI resources on parallel runs, so we've In order to maintain fast and reliable feedback, tests go through a promotion process. All new test cases or test suites must be labeled with the `@unstable` annotation. The Open MCT dev team runs these unstable tests in our private repos to ensure they work downstream and are reliable. -To run the stable tests, use the ```npm run test:e2e:stable``` command. To run the new and flaky tests, use the ```npm run test:e2e:unstable``` command. +- To run the stable tests, use the `npm run test:e2e:stable` command. +- To run the new and flaky tests, use the `npm run test:e2e:unstable` command. A testcase and testsuite are to be unmarked as @unstable when: @@ -293,13 +300,24 @@ Skipping based on browser version (Rarely used): } + */ +async function getCanvasPixels(page, canvasSelector) { + const getTelemValuePromise = new Promise(resolve => page.exposeFunction('getCanvasValue', resolve)); + const canvasHandle = await page.evaluateHandle((canvas) => document.querySelector(canvas), canvasSelector); + const canvasContextHandle = await page.evaluateHandle(canvas => canvas.getContext('2d'), canvasHandle); + + await waitForPlotsToRender(page); + await page.evaluate(([canvas, ctx]) => { + // The document canvas is where the plot points and lines are drawn. + // The only way to access the canvas is using document (using page.evaluate) + /** @type {ImageData} */ + const data = ctx.getImageData(0, 0, canvas.width, canvas.height).data; + /** @type {number[]} */ + const imageDataValues = Object.values(data); + /** @type {PlotPixel[]} */ + const plotPixels = []; + // Each pixel consists of four values within the ImageData.data array. The for loop iterates by multiples of four. + // The values associated with each pixel are R (red), G (green), B (blue), and A (alpha), in that order. + for (let i = 0; i < imageDataValues.length;) { + if (imageDataValues[i] > 0) { + plotPixels.push({ + r: imageDataValues[i], + g: imageDataValues[i + 1], + b: imageDataValues[i + 2], + a: imageDataValues[i + 3], + strValue: `rgb(${imageDataValues[i]}, ${imageDataValues[i + 1]}, ${imageDataValues[i + 2]}, ${imageDataValues[i + 3]})` + }); + } + + i = i + 4; + } + + window.getCanvasValue(plotPixels); + }, [canvasHandle, canvasContextHandle]); + + return getTelemValuePromise; +} + // eslint-disable-next-line no-undef module.exports = { createDomainObjectWithDefaults, createNotification, - expandTreePaneItemByName, - expandEntireTree, createPlanFromJSON, - openObjectTreeContextMenu, + expandEntireTree, + expandTreePaneItemByName, + getCanvasPixels, getHashUrlToDomainObject, getFocusedObjectUuid, + openObjectTreeContextMenu, setFixedTimeMode, setRealTimeMode, setStartOffset, setEndOffset, - selectInspectorTab + selectInspectorTab, + waitForPlotsToRender }; diff --git a/e2e/tests/functional/plugins/plot/overlayPlot.e2e.spec.js b/e2e/tests/functional/plugins/plot/overlayPlot.e2e.spec.js index bc27c5ef0a..818473ae15 100644 --- a/e2e/tests/functional/plugins/plot/overlayPlot.e2e.spec.js +++ b/e2e/tests/functional/plugins/plot/overlayPlot.e2e.spec.js @@ -26,7 +26,7 @@ necessarily be used for reference when writing new tests in this area. */ const { test, expect } = require('../../../../pluginFixtures'); -const { createDomainObjectWithDefaults, selectInspectorTab } = require('../../../../appActions'); +const { createDomainObjectWithDefaults, getCanvasPixels, selectInspectorTab, waitForPlotsToRender } = require('../../../../appActions'); test.describe('Overlay Plot', () => { test.beforeEach(async ({ page }) => { @@ -52,14 +52,9 @@ test.describe('Overlay Plot', () => { await page.locator('li.c-tree__item.menus-to-left .c-disclosure-triangle').click(); await page.locator('.c-click-swatch--menu').click(); await page.locator('.c-palette__item[style="background: rgb(255, 166, 61);"]').click(); - // gets color for swatch located in legend - const element = await page.waitForSelector('.plot-series-color-swatch'); - const color = await element.evaluate((el) => { - return window.getComputedStyle(el).getPropertyValue('background-color'); - }); - - expect(color).toBe('rgb(255, 166, 61)'); + const seriesColorSwatch = page.locator('.gl-plot-label > .plot-series-color-swatch'); + await expect(seriesColorSwatch).toHaveCSS('background-color', 'rgb(255, 166, 61)'); }); test('Limit lines persist when series is moved to another Y Axis and on refresh', async ({ page }) => { @@ -215,66 +210,26 @@ test.describe('Overlay Plot', () => { await page.goto(overlayPlot.url); // Wait for plot series data to load and be drawn - await expect(page.locator('.js-series-data-loaded')).toBeVisible(); + await waitForPlotsToRender(page); await page.click('button[title="Edit"]'); await selectInspectorTab(page, 'Elements'); await page.locator(`#inspector-elements-tree >> text=${swgA.name}`).click(); - // Wait for "View Large" plot series data to load and be drawn - await expect(page.locator('.c-overlay .js-series-data-loaded')).toBeVisible(); - - const plotPixelSize = await getCanvasPixelsWithData(page); + const plotPixels = await getCanvasPixels(page, '.js-overlay canvas'); + const plotPixelSize = plotPixels.length; expect(plotPixelSize).toBeGreaterThan(0); }); }); /** - * @param {import('@playwright/test').Page} page - */ -async function getCanvasPixelsWithData(page) { - const getTelemValuePromise = new Promise(resolve => page.exposeFunction('getCanvasValue', resolve)); - - await page.evaluate(() => { - // The document canvas is where the plot points and lines are drawn. - // The only way to access the canvas is using document (using page.evaluate) - let data; - let canvas; - let ctx; - canvas = document.querySelector('.js-overlay canvas'); - ctx = canvas.getContext('2d'); - data = ctx.getImageData(0, 0, canvas.width, canvas.height).data; - const imageDataValues = Object.values(data); - let plotPixels = []; - // Each pixel consists of four values within the ImageData.data array. The for loop iterates by multiples of four. - // The values associated with each pixel are R (red), G (green), B (blue), and A (alpha), in that order. - for (let i = 0; i < imageDataValues.length;) { - if (imageDataValues[i] > 0) { - plotPixels.push({ - startIndex: i, - endIndex: i + 3, - value: `rgb(${imageDataValues[i]}, ${imageDataValues[i + 1]}, ${imageDataValues[i + 2]}, ${imageDataValues[i + 3]})` - }); - } - - i = i + 4; - - } - - window.getCanvasValue(plotPixels.length); - }); - - return getTelemValuePromise; -} - -/** - * + * Asserts that limit lines exist and are visible * @param {import('@playwright/test').Page} page */ async function assertLimitLinesExistAndAreVisible(page) { // Wait for plot series data to load - await expect(page.locator('.js-series-data-loaded')).toBeVisible(); + await waitForPlotsToRender(page); // Wait for limit lines to be created await page.waitForSelector('.js-limit-area', { state: 'attached' }); const limitLineCount = await page.locator('.c-plot-limit-line').count(); diff --git a/e2e/tests/functional/plugins/plot/plotLegendSwatch.e2e.spec.js b/e2e/tests/functional/plugins/plot/plotLegendSwatch.e2e.spec.js deleted file mode 100644 index 2d3e2268f7..0000000000 --- a/e2e/tests/functional/plugins/plot/plotLegendSwatch.e2e.spec.js +++ /dev/null @@ -1,113 +0,0 @@ -/***************************************************************************** - * Open MCT, Copyright (c) 2014-2023, United States Government - * as represented by the Administrator of the National Aeronautics and Space - * Administration. All rights reserved. - * - * Open MCT is licensed under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * http://www.apache.org/licenses/LICENSE-2.0. - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations - * under the License. - * - * Open MCT includes source code licensed under additional open source - * licenses. See the Open Source Licenses file (LICENSES.md) included with - * this source code distribution or the Licensing information page available - * at runtime from the About dialog for additional information. - *****************************************************************************/ - -/* -Tests to verify log plot functionality. Note this test suite if very much under active development and should not -necessarily be used for reference when writing new tests in this area. -*/ - -const { selectInspectorTab } = require('../../../../appActions'); -const { test, expect } = require('../../../../pluginFixtures'); - -test.describe('Legend color in sync with plot color', () => { - test('Testing', async ({ page }) => { - await makeOverlayPlot(page); - - // navigate to plot series color palette - await page.click('.l-browse-bar__actions__edit'); - await selectInspectorTab(page, 'Config'); - - await page.locator('li.c-tree__item.menus-to-left .c-disclosure-triangle').click(); - await page.locator('.c-click-swatch--menu').click(); - await page.locator('.c-palette__item[style="background: rgb(255, 166, 61);"]').click(); - - // gets color for swatch located in legend - const element = await page.waitForSelector('.plot-series-color-swatch'); - const color = await element.evaluate((el) => { - return window.getComputedStyle(el).getPropertyValue('background-color'); - }); - - expect(color).toBe('rgb(255, 166, 61)'); - }); -}); - -async function saveOverlayPlot(page) { - // save overlay plot - await page.locator('text=Snapshot Save and Finish Editing Save and Continue Editing >> button').nth(1).click(); - - await Promise.all([ - page.locator('text=Save and Finish Editing').click(), - //Wait for Save Banner to appear - page.waitForSelector('.c-message-banner__message') - ]); - //Wait until Save Banner is gone - await page.locator('.c-message-banner__close-button').click(); - await page.waitForSelector('.c-message-banner__message', { state: 'detached' }); -} - -async function makeOverlayPlot(page) { - // fresh page with time range from 2022-03-29 22:00:00.000Z to 2022-03-29 22:00:30.000Z - await page.goto('/', { waitUntil: 'networkidle' }); - - // create overlay plot - - await page.locator('button.c-create-button').click(); - await page.locator('li[role="menuitem"]:has-text("Overlay Plot")').click(); - await Promise.all([ - page.waitForNavigation({ waitUntil: 'networkidle'}), - page.locator('button:has-text("OK")').click(), - //Wait for Save Banner to appear - page.waitForSelector('.c-message-banner__message') - ]); - //Wait until Save Banner is gone - await page.locator('.c-message-banner__close-button').click(); - await page.waitForSelector('.c-message-banner__message', { state: 'detached'}); - - // save the overlay plot - - await saveOverlayPlot(page); - - // create a sinewave generator - - await page.locator('button.c-create-button').click(); - await page.locator('li[role="menuitem"]:has-text("Sine Wave Generator")').click(); - - // Click OK to make generator - - await Promise.all([ - page.waitForNavigation({ waitUntil: 'networkidle'}), - page.locator('button:has-text("OK")').click(), - //Wait for Save Banner to appear - page.waitForSelector('.c-message-banner__message') - ]); - //Wait until Save Banner is gone - await page.locator('.c-message-banner__close-button').click(); - await page.waitForSelector('.c-message-banner__message', { state: 'detached'}); - - // click on overlay plot - - await page.locator('text=Open MCT My Items >> span').nth(3).click(); - await Promise.all([ - page.waitForNavigation(), - page.locator('text=Unnamed Overlay Plot').first().click() - ]); -} diff --git a/e2e/tests/functional/plugins/plot/plotRendering.e2e.spec.js b/e2e/tests/functional/plugins/plot/plotRendering.e2e.spec.js index 8587b035cb..a89c2f2d3b 100644 --- a/e2e/tests/functional/plugins/plot/plotRendering.e2e.spec.js +++ b/e2e/tests/functional/plugins/plot/plotRendering.e2e.spec.js @@ -26,26 +26,25 @@ */ const { test, expect } = require('../../../../pluginFixtures'); -const { createDomainObjectWithDefaults} = require('../../../../appActions'); +const { createDomainObjectWithDefaults, getCanvasPixels } = require('../../../../appActions'); -test.describe('Plot Integrity Testing @unstable', () => { +test.describe('Plot Rendering', () => { let sineWaveGeneratorObject; test.beforeEach(async ({ page }) => { - //Open a browser, navigate to the main page, and wait until all networkevents to resolve + // Open a browser, navigate to the main page, and wait until all networkevents to resolve await page.goto('./', { waitUntil: 'domcontentloaded' }); sineWaveGeneratorObject = await createDomainObjectWithDefaults(page, { type: 'Sine Wave Generator' }); }); test('Plots do not re-request data when a plot is clicked', async ({ page }) => { - //Navigate to Sine Wave Generator + // Navigate to Sine Wave Generator await page.goto(sineWaveGeneratorObject.url); - //Click on the plot canvas + // Click on the plot canvas await page.locator('canvas').nth(1).click(); - //No request was made to get historical data + // No request was made to get historical data const createMineFolderRequests = []; page.on('request', req => { - // eslint-disable-next-line playwright/no-conditional-in-test createMineFolderRequests.push(req); }); expect(createMineFolderRequests.length).toEqual(0); @@ -56,7 +55,8 @@ test.describe('Plot Integrity Testing @unstable', () => { await editSineWaveToUseInfinityOption(page, sineWaveGeneratorObject); //Get pixel data from Canvas - const plotPixelSize = await getCanvasPixelsWithData(page); + const plotPixels = await getCanvasPixels(page, 'canvas'); + const plotPixelSize = plotPixels.length; expect(plotPixelSize).toBeGreaterThan(0); }); }); @@ -70,70 +70,19 @@ test.describe('Plot Integrity Testing @unstable', () => { */ async function editSineWaveToUseInfinityOption(page, sineWaveGeneratorObject) { await page.goto(sineWaveGeneratorObject.url); - // Edit LAD table + // Edit SWG properties to include infinity values await page.locator('[title="More options"]').click(); await page.locator('[title="Edit properties of this object."]').click(); - // Modify the infinity option to true - const infinityInput = page.locator('[aria-label="Include Infinity Values"]'); - await infinityInput.click(); + await page.getByRole('switch', { + name: "Include Infinity Values" + }).check(); - // Click OK button and wait for Navigate event - await Promise.all([ - page.waitForLoadState(), - page.click('[aria-label="Save"]'), - // Wait for Save Banner to appear - page.waitForSelector('.c-message-banner__message') - ]); + await page.getByRole('button', { + name: 'Save' + }).click(); // FIXME: Changes to SWG properties should be reflected on save, but they're not? // Thus, navigate away and back to the object. await page.goto('./#/browse/mine'); await page.goto(sineWaveGeneratorObject.url); - - await page.locator('c-progress-bar c-telemetry-table__progress-bar').waitFor({ - state: 'hidden' - }); - - // FIXME: The progress bar disappears on series data load, not on plot render, - // so wait for a half a second before evaluating the canvas. - // eslint-disable-next-line playwright/no-wait-for-timeout - await page.waitForTimeout(500); -} - -/** - * @param {import('@playwright/test').Page} page - */ -async function getCanvasPixelsWithData(page) { - const getTelemValuePromise = new Promise(resolve => page.exposeFunction('getCanvasValue', resolve)); - - await page.evaluate(() => { - // The document canvas is where the plot points and lines are drawn. - // The only way to access the canvas is using document (using page.evaluate) - let data; - let canvas; - let ctx; - canvas = document.querySelector('canvas'); - ctx = canvas.getContext('2d'); - data = ctx.getImageData(0, 0, canvas.width, canvas.height).data; - const imageDataValues = Object.values(data); - let plotPixels = []; - // Each pixel consists of four values within the ImageData.data array. The for loop iterates by multiples of four. - // The values associated with each pixel are R (red), G (green), B (blue), and A (alpha), in that order. - for (let i = 0; i < imageDataValues.length;) { - if (imageDataValues[i] > 0) { - plotPixels.push({ - startIndex: i, - endIndex: i + 3, - value: `rgb(${imageDataValues[i]}, ${imageDataValues[i + 1]}, ${imageDataValues[i + 2]}, ${imageDataValues[i + 3]})` - }); - } - - i = i + 4; - - } - - window.getCanvasValue(plotPixels.length); - }); - - return getTelemValuePromise; } diff --git a/e2e/tests/functional/plugins/plot/tagging.e2e.spec.js b/e2e/tests/functional/plugins/plot/tagging.e2e.spec.js index 7ea739f24a..15f7ec40d9 100644 --- a/e2e/tests/functional/plugins/plot/tagging.e2e.spec.js +++ b/e2e/tests/functional/plugins/plot/tagging.e2e.spec.js @@ -25,7 +25,7 @@ Tests to verify plot tagging functionality. */ const { test, expect } = require('../../../../pluginFixtures'); -const { createDomainObjectWithDefaults, setRealTimeMode, setFixedTimeMode } = require('../../../../appActions'); +const { createDomainObjectWithDefaults, setRealTimeMode, setFixedTimeMode, waitForPlotsToRender } = require('../../../../appActions'); test.describe('Plot Tagging', () => { /** @@ -133,12 +133,9 @@ test.describe('Plot Tagging', () => { await expect(page.getByText('No results found')).toBeVisible(); //Reload Page - await Promise.all([ - page.reload(), - page.waitForLoadState('networkidle') - ]); + await page.reload({ waitUntil: 'domcontentloaded' }); // wait for plots to load - await expect(page.locator('.js-series-data-loaded')).toBeVisible(); + await waitForPlotsToRender(page); await page.getByText('Annotations').click(); await expect(page.getByText('No tags to display for this item')).toBeVisible();