Guide for Cross Platform E2E Test For Native Mobile APP

This article is a must-read for those experiencing headaches with mobile automation testing in the context of Continuous Integration and Continuous Deployment (CICD), particularly when it involves native mobile Apps for Android and iOS. It’s quite challenging to find sufficient resources that cover this specific topic.

Within this article, I will guide you through a detailed step-by-step process on how to create a comprehensive end-to-end test pipeline at no cost, utilizing GitHub Actions for both the iOS and Android platforms. We will be using our beloved WebdriverIO framework throughout the tutorial.

Challenge

Our challenge is to establish a unified pipeline workflow that enables testing of our native mobile application on both the iOS and Android platforms. In a previous article, we thoroughly explored the process of constructing a pipeline for testing an Android app using an emulator with GitHub Actions. To handle the E2E test for the Android component, we will reuse that workflow. However, we still need to address the remaining bigger challenge of creating a separate job for the iPhone/iPad simulator.

During my research, I came across an incredibly useful but often overlooked feature which is, the macOS GitHub runner comes pre-equipped with Xcode, the necessary SDKs for iPhone iOS simulators. This realization sparked an idea in my mind: why not replicate the process we followed for the Android emulator but this time for the iOS simulator? This is particularly exciting because GitHub Actions’ iOS runner supports virtualization, making it a viable option for our purposes.

This feature allows us to build our pipeline without any additional costs (up to a maximum of 2000 minutes).

Workflow structure

Our GitHub actions workflow will be basically the same as Android in the core concept but with a little technical variation

• Create simulator
• Install dependencies
• Execute the test
• Generate the report

Looks pretty straightforward, but how? Let’s see

Step 1

As previously mentioned, the GH Actions runner comes packed with a range of available simulators. While we could utilize one of these existing simulators, it would require using the deviceName and it's

randomly changing UUID for each execution. However, you can still extract the relevant UUID using shell commands.

To simplify the process and increase flexibility, we will create our own simulator. Since Xcode is already installed, we can make use of the “xcrun” CLI. To create a simulator from the installed iOS versions using the terminal, simply execute the following command:

xcrun simctl create "iPhone 14 Pro" "com.apple.CoreSimulator.SimDeviceType.iPhone-14-Pro" "com.apple.CoreSimulator.SimRuntime.iOS-16-0"

Executing this command will result in the immediate creation of a simulator and the subsequent retrieval of its UUID.

To enhance re-usability and optimize the process, we can encapsulate this command within a shell script. With a few modifications, we can ensure that the UUID is stored as an environment variable in GitHub Runner which we will eventually use for our test capabilities.

#!/bin/bash

# Set iPhone model and iOS version
iphone_model="${IPHONE_MODEL// /-}"
ios_version="${IOS_VERSION//./-}"
simulator_name="${iphone_model}"
simulator_udid=$(xcrun simctl create "$IPHONE_MODEL" "com.apple.CoreSimulator.SimDeviceType.$iphone_model" "com.apple.CoreSimulator.SimRuntime.iOS-$ios_version")

# Export the simulator UDID as an environment variable
export SIMULATOR_UDID="$simulator_udid"
echo "SIMULATOR_UDID=$SIMULATOR_UDID" >> $GITHUB_ENV

# Boot the simulator
xcrun simctl boot "$simulator_udid"

By using the script above, we can provide the device model and iOS version as environment variables which can be stored in the environment sections in our workflow, this will create the simulator and store its UUID in GITHUB_ENV. This UUID will be essential for configuring the desired capabilities in our tests.

Since we are using IPHONE_MODEL and IOS_VERSION as environment variable in our shell script then we will have to set them in the environment section as shown below.

Step 2

After successfully creating and booting up the simulator in the previous step, it’s crucial to verify that the process was completed without any issues and that the device is fully prepared for use.

To ensure the successful starting of our test, it is crucial to confirm that the IOS has fully booted. For this purpose, I have created a code snippet that continuously monitors the device’s status until a specific output is obtained, signifying the completion of the simulator’s booting process.

#!/bin/zsh

function wait_for_boot() {
  printf "${G}==> ${BL}Waiting for the simulator to boot...${NC}\n"
  start_time=$(date +%s)
  spinner=( "⠹" "⠺" "⠼" "⠶" "⠦" "⠧" "⠇" "⠏" )
  i=0
  # Get the timeout value from the environment variable or use the default value of 60 seconds
  timeout=${BOOT_TIMEOUT:-60}

  while true; do
    output=$(xcrun simctl bootstatus "$SIMULATOR_UDID")
    echo "${output}"
    if [[ $output == *"Device already booted, nothing to do."* ]]; then
      printf "\e[K${G}==> \u2713 Simulator booted successfully${NC}\n"
      exit 0
    else
      printf "${YE}==> Please wait ${spinner[$i]} ${NC}\r"
      i=$(( (i+1) % 8 ))
    fi

    elapsed_time=$(( $(date +%s) - $start_time ))
    if [[ $elapsed_time -ge $timeout ]]; then
      printf "${RED}==> Timeout waiting for simulator to boot 🕛${NC}\n"
      exit 1
    fi

    sleep 1
  done
}

# Call the wait_for_boot function
wait_for_boot

Step 3

Proceeding further, we will cover the necessary steps and dependencies required for executing your tests. This includes the installation of Appium, the XCUITest driver, and the essential Node.js libraries.

  "devDependencies": {
    "@wdio/allure-reporter": "^8.10.4",
    "@wdio/appium-service": "^8.10.5",
    "@wdio/cli": "^8.10.5",
    "@wdio/local-runner": "^8.10.5",
    "@wdio/mocha-framework": "8.10.4",
    "@wdio/spec-reporter": "8.8.7",
    "ts-node": "^10.9.1",
    "typescript": "^5.0.4"
  },
  "dependencies": {
    "allure-commandline": "^2.22.1",
    "appium": "2.0.0-beta.71",
    "appium-uiautomator2-driver": "*",
    "appium-xcuitest-driver": "*"
  }

Connecting the Puzzle Pieces

Since now the key elements required to set up the environment for executing our mobile automation tests on the iOS simulator are ready, Let’s wrap them all into a single yaml file for GH actions

name: Wdio-x-native

on:
  workflow_dispatch:

env:
  IPHONE_MODEL: iPhone 8
  IOS_VERSION: 16.2
  BOOT_TIMEOUT: 700

jobs:
  ios:
    runs-on: macos-13

    steps:
      - uses: actions/checkout@v3

      - name: Export environment variables
        run: |
          export IPHONE_MODEL=$IPHONE_MODEL
          export IOS_VERSION=$IOS_VERSION

      - name: Start simulator
        run: |
          chmod a+x ./sscript/start_simu.sh
          ./sscript/start_simu.sh

      - name: Install dependencies
        run: |
          npm i

      - name: Check simulator booting status
        run: |
          chmod a+x ./check_simu.sh
          ./check_simu.sh

      - name: Execute the test
        run: |
          npm run ios

Combine the simulator status check and simulator start into a single shell script would have been possible. However, I intentionally separated them to execute them individually. This allows me to utilize the time taken for the simulator to boot up and install the remaining dependencies. Afterwards, I can then proceed to check the status of the simulator. Similarly we will apply same approach to Android emulator (check previous article).

Build cross-platform workflow

The time now to combine our Android workflow from the previous article without Ios workflow into one single workflow, using the matrix strategy as follows:

name: Wdio-x-native

on:
  workflow_dispatch:

env:
  IPHONE_MODEL: iPhone 8
  IOS_VERSION: 16.2
  API_LEVEL: 32
  EMULATOR_NAME: Nexus
  EMULATOR_DEVICE: Nexus 5
  EMULATOR_VERSION: 12
  ANDROID_ARCH: x86_64
  ANDROID_TARGET: google_apis
  ANDROID_BUILD_TOOLS_VERSION: 34.0.0-rc4
  ANDROID_SDK_PACKAGES: system-images;android-32;google_apis;x86_64 platforms;android-32 build-tools;34.0.0-rc4 platform-tools emulator
  EMULATOR_TIMEOUT: 350
  BOOT_TIMEOUT: 700

jobs:
  ios:
    runs-on:
       - macos-13
    strategy:
      matrix:
        os: [IOS]
        device: [$IPHONE_MODEL]
        version: [$IOS_VERSION]
    steps:
      - uses: actions/checkout@v3

      - name: Export environment variables
        run: |
          export IPHONE_MODEL=$IPHONE_MODEL
          export IOS_VERSION=$IOS_VERSION
    # ...
    # find the full workflow at the end of the article
    # ...

  android:
    runs-on: macos-13
    strategy:
      matrix:
        os: [Android]
        emulator_name: [$EMULATOR_NAME]
    steps:
      - uses: actions/checkout@v3

      - name: Add avdmanager and sdkmanager to system PATH
        run: |
          echo "$ANDROID_HOME/cmdline-tools/latest/bin:$ANDROID_HOME/emulator:$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools:$ANDROID_HOME/build-tools/${{ env.ANDROID_BUILD_TOOLS_VERSION }}" >> $GITHUB_PATH

      - name: Install Sdk
        run: |
          yes Y | sdkmanager --licenses
          sdkmanager --install ${ANDROID_SDK_PACKAGES}

      - name: Build emulator

    # ...
    # find the full workflow at the end of the article
    # ...

In the above example, we have integrated the IOS workflow mentioned earlier with the Android emulator workflow described in our previous article.

These are the recommended configurations that you may require for both the Android emulator and iPhone simulator. It’s important to note that the deviceName, platformVersion, and UUID are not hardcoded in our object. This flexibility allows us to easily switch between different versions and device models as needed.

const emulator = [{
  platformName: 'android',
  'appium:options': {
    deviceName: process.env.CI ? process.env.EMULATOR_NAME : 'Nexus',
    platformVersion: process.env.CI ? process.env.EMULATOR_VERSION : '13',
    automationName: 'uiautomator2',
    appPackage: 'com.wdiodemoapp',
    appWaitPackage: 'com.wdiodemoapp',
    appActivity: 'com.wdiodemoapp.MainActivity',
    appWaitActivity: 'com.wdiodemoapp.MainActivity',
    uiautomator2ServerLaunchTimeout: 200000,
    uiautomator2ServerInstallTimeout: 200000,
    appWaitForLaunch: true,
    autoGrantPermissions: true,
    adbExecTimeout: 200000,
    androidInstallTimeout: 150000,
    ignoreHiddenApiPolicyError: true,
    noReset: true,
    fullReset: false
  }
}]

const simulator = [{
  platformName: 'iOS',
  'appium:options': {
    deviceName: process.env.CI ? process.env.IPHONE_MODEL : 'Iphone-13',
    platformVersion: process.env.CI ? process.env.IOS_VERSION : '15.5',
    automationName: 'XCUITest',
    bundleId: 'org.wdioNativeDemoApp',
    app: 'iOS-Simulator-NativeDemoApp-0.4.0.app.zip',
    udid: process.env.CI ? process.env.SIMULATOR_UDID : '15A098DB-B8A0-4D6A-9057-23FF1F0F0D9B',
    useNewWDA: true,
    usePrebuiltWDA: false,
    wdaConnectionTimeout: 180000,
    appWaitForLaunch: true,
    noReset: true,
    fullReset: false
  }
}]

Initial Execution

The good news is that workflow is configured properly and the e2e test for the IOS app has been successfully executed

Though the end-to-end test for the iPhone simulator has passed, it was observed that the test for the Android emulator shows instability.

Debugging

System UI Crush

It appears that running Android for the first time in headless mode occasionally results in random system UI unresponsive issues. Unfortunately, this issue is preventing us from executing the tests as the UI system is unresponsive which consequently disrupts Appium from proper interaction with the app.

The issue was confirmed when reviewing the allure report screenshots

This explains why the terminal log displayed that Appium was unable to locate any element, despite the successful launch of the app.

This makes sense as Appium is trying to find the desired element but on the current running activity which is .systemui, even though our target app is launched in the background

Connection timeout

It has been noted that on certain occasions, Appium encountered failures in initiating the test, with all connection retry attempts proving unsuccessful. However, after conducting a thorough investigation, it was discovered that the installation of the Apk file to the Android emulator through the app:“./test.apk” capability was taking an unusually long time, requiring a significantly extended connection timeout to ensure successful installation which is not the best solution.

Now that we have identified the issue and its root cause, it’s time to address and resolve them.

Resolving

System UI Crush

Fortunately, we can utilize the advantage of being able to grep the current running activity on an Android device. This privilege allows us to detect whether the system UI or any similar Android service will crash or function normally. We can achieve this by executing the following adb shell command:

adb shell dumpsys window 2>/dev/null | grep -i mCurrentFocus

In our ongoing implementation, we can mimic our natural behavior when encountering this issue on an Android device. Specifically, we will continuously click the home button until the issue is resolved. Once the problem is resolved and the Android system is functioning correctly, we anticipate observing the “.NexusLauncherActivity” as the current main activity running (where “Nexus” represents the Android device).

In order to achieve this, I have developed the following shell script:

#!/bin/bash

function check_current_focus() {
  printf "==> Checking emulator running activity \n"
  start_time=$(date +%s)
  i=0
  timeout=20
  target="com.google.android.apps.nexuslauncher.NexusLauncherActivity"

  while true; do
    result=$(adb shell dumpsys window 2>/dev/null | grep -i mCurrentFocus)

    if [[ $result == *"$target"* ]]; then
      printf "==>  Activity is okay: \n"
      printf "$result\n"
      break
    else
      adb shell input keyevent KEYCODE_HOME
      printf "==> Menu button is pressed \n"
      i=$(( (i+1) % 8 ))
    fi

    current_time=$(date +%s)
    elapsed_time=$((current_time - start_time))
    if [ $elapsed_time -gt $timeout ]; then
      printf "==> Timeout after ${timeout} seconds elapsed 🕛.. \n"
      return 1
    fi
    sleep 4
  done
}

check_current_focus

The shown function above will continuously loop, If the main activity is not found (NexusLauncherActivity), it will send a home button event and repeat the process until its found or timeout is reached.

Connection timeout

Rather than significantly extending the Appium connection timeout, I will handle the APK installation in a separate step along with the main activity checking.

      - name: Install APK
        run: |
          adb install Android-NativeDemoApp-0.4.0.apk
          chmod a+x ./mainActivityCheck.sh
          ./mainActivityCheck.sh

Excellent! Our solution has been executed successfully with the proper installation of the APK. As expected, the system UI was not responsive, and the shell script effectively managed and handled the situation.

Optimizing and Enhancing workflow

I have improved the workflow dispatch to provide better control over the platforms on which I can execute my tests, whether it be iOS, Android, or cross-platform.

name: Wdio-x-native

on:
  workflow_dispatch:
    inputs:
      e2e:
         type: choice
         description: Select a platform
         required: true
         options:
            - xplatform
            - ios
            - android
         default: xplatform

Consequently, our jobs should be adjusted

name: Wdio-x-native

on:
  workflow_dispatch:
    inputs:
      e2e:
         type: choice
         description: Select a platform
         required: true
         options:
            - xplatform
            - ios
            - android
         default: xplatform


permissions:
  contents: write
  pages: write
  id-token: write

env:
  IPHONE_MODEL: iPhone 8
  IOS_VERSION: 16.2
  API_LEVEL: 32
  EMULATOR_NAME: Nexus
  EMULATOR_DEVICE: Nexus 5
  EMULATOR_VERSION: 12
  ANDROID_ARCH: x86_64
  ANDROID_TARGET: google_apis
  ANDROID_BUILD_TOOLS_VERSION: 34.0.0-rc4
  ANDROID_SDK_PACKAGES: system-images;android-32;google_apis;x86_64 platforms;android-32 build-tools;34.0.0-rc4 platform-tools emulator
  EMULATOR_TIMEOUT: 350
  BOOT_TIMEOUT: 700

jobs:
  ios:
    runs-on:
       - macos-13
    if: ${{ contains(github.event.inputs.e2e, 'ios') || contains(github.event.inputs.e2e, 'xplatform') }}
    strategy:
      matrix:
        os: [IOS]
        device: [$IPHONE_MODEL]
        version: [$IOS_VERSION]
    steps:
      - uses: actions/checkout@v3
    # find the full workflow at the end of the article

  android:
    runs-on: macos-13
    if: ${{ contains(github.event.inputs.e2e, 'android') || contains(github.event.inputs.e2e, 'xplatform') }}
    strategy:
      matrix:
        os: [Android]
        emulator_name: [$EMULATOR_NAME]
    steps:
      - uses: actions/checkout@v3
    # find the full workflow at the end of the article

Finally, Generate our report and deploy it to the GitHub page out of the box

      - name: Generate report
        if: always()
        run: |
          npx allure generate report/allure-results

      - name: Setup Pages
        if: always()
        uses: actions/configure-pages@v3

      - name: Upload artifact
        if: always()
        uses: actions/upload-pages-artifact@v1
        with:
          path: './allure-report'

      - name: Deploy to GitHub Pages
        if: always()
        id: deployment
        uses: actions/deploy-pages@v2

Workflow Execution

Fantastic news! Our workflow is now functioning flawlessly and exhibits complete stability. The workflow can be triggered against a single platform, such as Android or iOS, or simultaneously against both platforms in parallel.

Full workflow

loading...

View on GitHub

Conclusion

By leveraging the capabilities provided by GitHub Actions, which offer out-of-the-box SDKs for both Android and iOS, we gain a significant advantage. This allows us to construct an efficient end-to-end test pipeline without incurring any costs or relying on mobile device farm cloud services. Although conducting tests on real devices is preferable, particularly for Android, the cost-free nature of this approach presents a satisfactory compromise.

Throughout our discussion, we have provided a step-by-step demonstration of building a cross-platform end-to-end test using the GitHub Actions pipeline for native mobile Apps. We have addressed various challenges, obstacles, and issues, ensuring a thorough understanding of the process. Armed with this knowledge, you should find it easier to construct your own customized pipeline that caters to your specific requirements.