Last year, I made the promise we would combine pipelines with test automation. Unfortunately, this feature is not available, yet. The good news however is that recently the Power Platform release wave 1 has been published. On the roadmap we can see that the Power Apps CLI will be updated to support Test Engine, which allows us to implement tests in CI/CD processes. This is all expected in May this year.
That's why I decided to rebrand it to the Power Platform Testing Trilogy. Our first saga was on Test Studio. The second saga, which is this challenge, we will dive into Test Engine. Once the CLI is updated, you can expect the closing piece of this journey.
Challenge Objectives
🎯 Create a local build of Test Engine
🎯 Run a sample test
🎯 Re-use the test created during The Fellowship of Test Automation
🎯 Be fully prepared for the climax of this trilogy
🤓 What is Test Engine?
We all love to build new functionality, and improve apps that we've already built. But we also know that testing is an important aspect of software development. A newer version should add new functionality to your app, but also should keep earlier implemented features functional.
Power Apps Test Studio has been introduced earlier, for exactly the same reason. However, Test Engine brings it even further. It uses Playwright to orchestrate the tests. Some key features:
The feature that I am most excited about, is that tests are authored with Power Fx, the language you use in Canvas Apps. This enables us, the non-code first types, to level up our app building game.
Another very cool feature is called connector mocking. This basically means that you can execute your tests, without interfering with your live data sources.
You can add the function Screenshot() during different states of your app. This is a useful tool for trouble shooting. This way you can use these images directly without reproducing the test.
We can make these test part of our CI/CD system. This way we can run the tests before we deploy a solution to a production environment.
This challenge will introduce you to all the advantages mentioned above.
Getting Started
Just like we did with code components, you will take the first steps from documentation that already exists. Below I will guide you through what I did.
Build locally
If you have finished the code components challenge, you probably have the given prerequisites already fixed. If not, make sure you have all things installed. How to fix the environment variable is clearly described in the challenge mentioned, so please read how to do this there.
Below I have copied the PowerShell commands from the documentation. Before you clone the GitHub repo, I suggest you open a folder where you want it cloned to in Visual Studio code. You can start a Terminal within Visual Studio Code, which is quite convenient. For the location you can pick your desktop, a dedicated GitHub folder, or any other location you like. I have a dedicated GitHub folder in my OneDrive, so I can get there on any device.
If the dotnet build function is not working, you could run msbuild. This is also explained in the code components challenge.
With the clone git operation, you downloaded a PowerShell script called playwright.ps1. This contains some commands to install the browsers that Playwright uses for running the tests. You need to specify the relative path of the file. You can use the Explorer within Visual Studio Code for this (see image below).
# Clone GitHub repo
git clone https://github.com/microsoft/PowerApps-TestEngine.git
# Change to the PowerAppsTestEngine foldercd PowerApps-TestEngine\src\PowerAppsTestEngine
# Build
dotnet build
# Install browsers required by Playwright - replace <net-version> with actual output folder name, eg. net6.0.
.\bin\Debug\<net-version>\playwright.ps1 install
Sample solution
We now have all the required files to work with. The documentation now suggests to use one of the provided samples. I used the first option, the basic gallery. You can import this solution to your environment of choice. If you want to know how to do it with the PAC CLI as suggested by the documentation, come back next month (just a minor hint).
Config file
The next step is provide some properties in the config.dev.json file. Just copy/paste the provided config file and rename it accordingly. Below is described where these properties can be found.
environmentid. Go to aka.ms/ppac, select Environments on the left navigation pane, select the environment wher you've imported the basic gallery solution. Within the details you can find the Environment ID.
tenantid. Go to portal.azure.com, navigate to Azure Active Directory. On the Overview screen you can copy the Tenant ID at the Basic information section.
testPlanFile. you can past in the example text from the documentation: ../../samples/basicgallery/testPlan.fx.yaml. You can find this exact file again by navigating through the explorer. Note that ../../ of the test means you navigate two levels up from where the config file is stored.
The other fields are optional and we'll leave them blank for now.
User authentication
The documentation is not really elaborate on how to deal with credentials. That's why I will explain a bit. If you open the file we specified at the testPlanFile property from the last section in Visual Studio Code, you can see that at the bottom it uses environmentVariables for the credetials. The emailKey and passwordKey is input for Test Engine. These are mapped to user1Email and user1Password, which should be environment variables.
We don't have these yet. The linked documentation is a bit more clear on this. you can do that by copying the lines below into the terminal and hit enter. Obviously, you have to replace as indicated, including the <>.
$env:user1Email = "<replace with your email>"
$env:user1Password = "<replace with your password>"the equal sign is PowerShell syntax for setting s variable. it is quite similar to Set() or UpdateContext() you might know from Power Fx. You can check the stored variable by just typing the variable (e.g. $env.user1Email) in the Terminal.
Now everything is in place, it's time to give it a spin locally. Run the command below in your terminal.
dotnet runMFA
Did you get an good result? I didn't. The problem that I had, and that is probably identical to your situation, is that it didn't proceed because of multi-factor authentication. The nice thing is that during the run, a new folder has been added to your project. Note that you ran the dotnet run operation from withing the PowerAppsTestEngine folder. Here you can find the new folder TestOutput. every run will add a subfolder. If you browse through that subfolder, you can see a folder that contains Basic Gallery. In here you can find a small recording of what actually happened. watch it to see that it doesnt proceed because of MFA.
The documentation states that Test Engine does not support MFA. Approving your sign-in with the Microsoft Authenticator is a form of MFA, which is probably something you use all the time.
Many businesses enforce MFA for any user. In a real-world scenario you should contact your security department to discuss the possibilities in your organization. A well-known solution is to set up a service account that does not require any form of MFA. Be aware that this service account should then also be added as a user to the app.
For now, we will do it the easy way. For security reasons, Microsoft enables security default for each tenant. That means that your developer tenant probably has this setting enabled too. For the sake of this challenge, we will turn it off. You can turn it back on after finishing this challenge, or try to create such a service account yourself.
Go to portal.azure.com
Go to Azure Active Directory
Select Properties on the left navigation pane
Click on Manage security defaults, and turn it off (enable, no)
Now, give it another try with the dotnet run command. It should now proceed a little further.
Localization
Depending on you location settings of your machine, you could run into another issue. While running, Test Engine prints it's progress to the terminal. This gives us some information for error handling. As you can see from the image below, my test case failed. You can see it directs towards a file that contains the test results. There you can review the error message.
In my case, the error message was Dutch and tells me an unexpected character was found, where a ParenClose was expected. This tells me there is probably some syntax issue. Luckily for us, Ben den Blanken already raised this issue on Github.
In a nutshell, the problem is that different regions use different characters for separating and chaining operations. If you open the link, you probably know what we need to fix. We need to change every : into ;; and every , into ; of the testPlanFile. We can use the find and replace function within Visual Studio Code for a quick fix. Hit Ctrl + F and toggle the replace function. Be aware of the sequence of the find and replace actions 😉.
Make sure to safe the testPlan file, and run your test again. You should now get all cases passed in case you are not in the same localization style as the US. This is work
Now that everything ran successfully, it's worthy to open the recording to see how the test was executed. The getting started section from Test Engine's GitHub is now finished. The What to do next section indicated that we can download recorded tests from Test Studio. This is what we did in challenge 11, so we will re-use that test to see how it works on our own creation.
Migrate Test Studio tests to Test Engine
I know you come here every month and have everything, including the output of challenge 011, saved like it is you life savings. I do encourage you to try it with your own creation. Just know that the file below is here for you for an easy start, or for trouble shooting if you really challenges yourself. In case you will use it, you obviously have to install it first.
If you either use the app you created earlier or install the app above, open it and go to Test Studio.
You can download the full test by pressing Download suite.
The next step is to add the downloaded test file to the Visual Studio Code project. You can either create a folder in Visual Studio code and drag the downloaded file to it, or add the folder and file using file explorer. The final output should be like the image below.
Have a look at the test file you've added. You can see it has a bit more context information in it. The screenWidth and screenHeight setting can cause some issues, so erase those. Also, the timeout property is set to 30000, which means the test will stop after 30 seconds. also remove this property.
Now it's time to update config.dev.json file. Make sure the right environment is selected and set the TestPlanFile to the newly added test file. In my case it's set to "../../challenge_011_test/Issues.fx.yaml".
That's it for a first run. Enter dotnet run in the terminal and run it to see the results.
As you can see, I ran into some issues. Maybe you did too. What I found out is that the export function is not yet operating as it should. If you compare the yaml test file with what is built in Test Studio, you can see the sequence is different. The test is running line by line. This is why only my Navigate test case is succeeding, as that only contains one action. Make sure to order everything as it is withing the Test Studio. Sorry for the inconvenience.
Now run the test again. Unfortunately, You probably still get some issues, just like I did. What seems to be the problem is that we set some variables that we later use in an Assert function, to determine if the functionality actually worked. But this is not a supported feature yet.
That's it for now. Just like every movie in which the story will continue has a cliffhanger, so does this challenge. Will the function be supported in time? Will exporting the suite be ordered properly? Will the localization work as it should? You will find out during the epic final of this journey.
Additional Information
At the moment, Test Engine only supports Canvas Apps. The release wave mentioned in the introduction also indicates model-driven apps and power pages will be supported later this year. In combination with the fact that localization is not supported yet and the CLI will support Test Engine, we can expect quite some development in this area for the coming months.
Key Takeaways
👉🏻 Test Engine is just the framework to execute the tests using Playwright
👉🏻 You can use the output of your Test Studio suite as input for your test plan file
👉🏻 Localization is not (yet) supported, so this might require minor adjustments
👉🏻 The output or the Test Studio suite is not ordered correctly. This must be done manually
👉🏻 Not all functions are supported yet
👉🏻 Knowledge of the PowerApps CLI (hint) will be a helpful skill for the closing piece
Comments