How to run fwupd tests with Moblab¶

This howto shows how to run the fwupd test suites on Moblab to verify the firmware updatability of peripherals in ChromeOS.

Overview¶

fwupd is a system daemon that allows an OS to update the firmware of a wide array of peripherals. ChromeOS relies on it to perform the firmware updates of compatible and updatable devices.

Moblab contains a set of fwupd test suites to test the basic firmware-related operations that fwupd can perform on a device. The purpose of these tests is, primarily, to validate the correct firmware updatability of new peripherals so that they comply with the WWCB Certification and to check the consistency and correctness of these operations across different ChromeOS versions and firmware releases.

Before you begin¶

To run the fwupd tests you’ll need a working Moblab setup with at least one DUT (a Chromebook or Chromebox) running a ChromeOS test image.

How to get a Partner Domain account¶

In order to have access to ChromeOS test releases you’ll need a Partner Domain account. To ask for an account, send a request to cros-pd-owners@google.com.

Note

Besides the Partner Domain account, you should also get access to a CPCon account, a Service account and a GCS bucket tied to the CPCon account.

How to ask for access to ChromeOS images for specific boards¶

Depending on the Chromebooks and Chromeboxes that you use for the tests, you’ll need to have explicit access to the ChromeOS test images for each specific model or board type so you can download them from the ChromeOS Partners Portal.

To request access, create an issue in the Partner Issue Tracker using Component “ChromeOS Public Tracker > Services > Infra > Moblab” (1038089) and template “Build Permission Access Request” and fill in the template details.

How to get a Moblab¶

Reference: Moblab Instruction Manual section “II - Requirements”

Moblab is based on a Google Chromebox (Wukong or Wyvern). The distributor of Moblab is CREATE TOGETHER TECHNOLOGY Co. Ltd. To inquire a quotation, reach out to:

Hans (hans@cttech-group.com)
Kiki (kiki@cttech-group.com)

Please note, Moblab is not supported in mainland China, nor has it been certified by China CCC.

Required hardware¶

One or more DUTs (Chromebook or Chromebox)
Network equipment as described in section “II - Requirements” of the Moblab Instruction Manual, at least two Ethernet cables and a USB to Ethernet dongle. Follow the instructions in section “II.2 - Now that I have all the required hardware, how do I connect them?” in the Moblab Instruction Manual to setup the test lab
A USB flash drive larger than 8GB
An hdmi or displayport monitor, a keyboard and a mouse
If the DUT is a Chromebox: an additional monitor, keyboard and mouse

Required software¶

Chrome Recovery Utility extension installed in Chrome.

Initial DUT (Chromebook) setup¶

Main references:

In order to use a Chromebook or Chromebox for Moblab tests it needs to be running a ChromeOS test release. Before continuing, make sure you have a partner domain account with access to the ChromeOS Partners Portal and to the ChromeOS images for the specific Chromebook/Chromebox board types you want to test.

Flash a ChromeOS test Image¶

First, start by flashing a ChromeOS test image into a USB flash drive:

Go to the ChromeOS Partners Portal <https://www.google.com/chromeos/partner/fe/#release> and log in with your partner domain account and click the “Image Files” tab.
Select the board of the Chromebook or Chromebox to test and TEST_IMAGE_ARCHIVE in the “Image Type” drop-down menu, and click “Search” to list all the ChromeOS test images for that board. You can refine the search by entering a specific “Release/Milestone” and/or a specific “Version/prefix”.
Download a recent test image (dev or stable channels recommended) and decompress it with:
Click the extension button in Chrome, then select Chrome Recovery Utility
Select the gear icon in the window. Next, click Use local image.
Select the chromium_test_image.bin file extracted in step 3
Plug in the USB flash drive and select it as the media to use. Click Continue and then Create now. Wait until the image is completely written to the USB drive.
Once complete, Select Done then unplug the USB flash drive

Install test image¶

Next, to install the image in the Chromebook or Chromebox, follow these steps:

Put the device into Developer Mode with the following procedure:
1. For Chromebooks, Hold Esc + Refresh and press the Power button. For Chromeboxes, engage the small Reset pinhole with a paperclip, hit Power and continue engaging Reset for 2 seconds. This will put the device into Recovery Mode and it should show a screen similar to this:
  
  Or this, depending on the model:
  
  Note
  
  On some Chromebooks the combination to hold is Esc + Full screen instead.
2. In the Recovery Mode screen, press Ctrl + D, followed by Enter to enter Developer Mode
  
  Note
  
  For other devices without keyboards (such as tablets) follow these instructions to enter Recovery Mode and Developer Mode
3. Wait until the process is done and the Developer Mode warning screen appears
Once the device is in Developer Mode, it will show the warning screen above every time it boots. It’ll start ChromeOS after 30 seconds or if you press Ctrl + D. Start ChromeOS and wait for it to show the welcome screen
Go to virtual terminal 2 to access a command line prompt by pressing:

where the [ → ] key is the right-arrow key just above the number 3 on the keyboard. If the keyboard doesn’t have this key, use the key in the F2 position. Then log in with user: root
Enable USB boot with the following commands:
Now reboot and wait for the Developer Mode warning screen to appear, plug in the USB flash drive and press Ctrl + U to boot the ChromeOS test image from the USB drive.

Wait for ChromeOS to start
Once ChromeOS is running, go to virtual terminal 2 again and log in with user: root and password: test0000. Then install the test image in the hard disk with the following command:

and follow the instructions
Once installation has completed, reboot the device (shutdown -h now) and remove the USB flash drive

Moblab setup¶

Follow the instructions in the Moblab Introduction & User Manual to configure the Moblab, connect the DUT (Chromebook or Chromebox) to it and enroll it. The end result must be something like this, where the “Manage DUTs” tab shows an enrolled DUT with a “Ready” Status:

Other considerations and requirements¶

About the peripheral status¶

The peripherals to test must be in working order. If they are meant to be updated wirelessly, they must have sufficient battery level to ensure the firmware update process can be completed successfully. They must also be supported by fwupd and they must have at least a firmware release included in the ChromeOS-specific fwupd remotes, which are defined in the latest .ebuild file in https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/main/sys-firmware/fwupd-peripherals/.

About the fwupd version and peripheral support¶

The fwupd version might be different from one ChromeOS version to another, so a device that is supported by fwupd in a newer ChromeOS version might not be supported in an older one.

About DUT provisioning in Moblab¶

Each Moblab test run will start by provisioning the DUT, that is, updating its ChromeOS version to the one specified by the tester and checking that it can run the tests properly. This has some side effects:

The DUT will constantly check for an Ethernet link and will reboot automatically if it doesn’t detect one after a few seconds.
All Bluetooth device pairings will be wiped out

A DUT can be also provisioned on demand by selecting the DUT in the “Manage DUTs” tab and clicking on the “Provision DUTs” button, then selecting the milestone and build to use:

Test cases¶

Note

The FWUPD tests are available only on ChromeOS R113-15382.0.0 and later.
Tests of firmware updates/downgrades/installs over Bluetooth links are not properly supported at the moment

The procedure to run the tests is the same for all the test cases except for the parameters they take. To start a test, follow these steps:

Make sure the DUT that will run the test is running, connected to the Moblab network and that it shows up in the “Manage DUTs” tab in Moblab
Enroll the DUT for tests by selecting it in the “Manage DUTs” tab and then clicking “Enroll Selected”
Make a note of the DUT IP address, as it’ll need to be specified later as one of the test parameters
Connect the peripheral to the DUT and power it on
Go to the “Run Suite” tab and select “FWUPD” in the top menu
Select the model and build target of the selected DUT, and then the ChromeOS milestone and build to run the test on. Important: only ChromeOS R113-15382.0.0 and later. The tests won’t be started for any version older than that.
Select the IP address of the DUT that will run the test

The rest of the steps depends on the test case to run:

Update a device firmware to the latest release¶

This test case will try to update a device firmware to the latest release available in the fwupd remotes. To run it:

Select the fwupd_update suite
Select the device you want to test
Click on “Run Suite”

Note

If there aren’t any new firmware releases available for the device, the test won’t proceed and will be marked as an error.

Downgrade a device firmware to the previous release¶

This test case will try to downgrade a device firmware to the previous release found in the fwupd remotes. To run it:

Select the fwupd_downgrade suite
Select the device you want to test
Click on “Run Suite”

Note

If there aren’t any previous firmware releases available for the device, the test won’t proceed and will be marked as an error.

Install a firmware version¶

This test allows the user to flash any available firmware release into a device, regardless of the current version running on it. To run it:

Select the fwupd_install_version suite
Select the device you want to test
Input the release version you want to install. Note that the version must be specified in the format defined by the hardware vendor (as a number pair, triplet, hexadecimal number, etc.). The Device info at the bottom shows the Current FW version with the expected format.
Click on “Run Suite”

Install a firmware file¶

This test allows the user to flash a specific firmware file into a device. The file can be provided either as a URL if it’s in a remote server, or through an external drive connected to the DUT.

If the file is provided in an external drive, a USB flash drive is recommended. Format it as FAT32 and set FWUPDTESTS as its label. Then copy the firmware file to it and plug it to the DUT.
If the file is provided as a URL, it must be accessible for the DUT to download.

To run the test:

Select the fwupd_install_file suite
Select the device you want to test
Enter the complete URL of the firmware file if it’s a remote file or the file name if it’s a file provided through a USB flash drive
Click on “Run Suite”

How to verify the test results¶

Once a test has started, it’ll show up in the “View Jobs” tab. You can enable the “Auto Refresh” toggle switch to keep the job list updated as the job status progress:

When the test finishes, its status will change to “COMPLETE”, the logs will be stored locally and eventually uploaded to a Google Storage bucket. A cloud icon in the “Logs” column means that the logs have been uploaded to Google Storage and are no longer stored locally.

Note that for each test run there’ll be two entries in the jobs table: one representing the test suite (control.fwupd_update in the image above) and another representing the test proper (fwupd_FirmwareUpdate in the image).

Once the job has finished you can check the results. If the logs haven’t been uploaded yet, you can check them directly by clicking on the “Logs” icon for the test job (not the suite). From there, navigate to the “status.log” file:

The “status.log” file shows a summary of the test result:

In this case it shows that the FirmwareUpdate test failed because the selected device doesn’t have any FW releases available.

This other case shows a successful run:

When the logs are uploaded to Google Cloud Storage, you can check them through CPCon, accessing with your partner domain account and clicking on the “Autotest View” option. From there you can see all the test suites run on every Chromebook type and ChromeOS milestone:

Clicking on any of the test suite results will lead you to a detailed summary of that suite. All the different test runs for that particular Chromebook type and ChromeOS version will show up listed either as a “Non-Passed test” or as a “Passing test”. The most recent test run will be the last log listed:

Clicking on any of the logs will direct you to the Google Storage bucket directory containing the logs for that test run:

From there you can download the “status.log” file. The directory and file structure is the same as in the locally stored logs.

How to get debug information¶

If the “status.log” report doesn’t give enough information about a failed test, you can also download the “debug/client.0.DEBUG” file, which contains the full log including debug messages. Additionally, the “sysinfo/messages” file contains the full system log that can also be useful to investigate a bug.

For instance, this “install file” test failed with this message in the status.log file:

Which doesn’t tell the reason. Checking the “debug/client.0.DEBUG” file shows the whole fwupdmgr output:

This means that the file used for the update isn’t compatible with the specified device.

FAQs¶

How to find the board type of a Chromebook or Chromebox?¶

In the Chromebook/Chromebox, open Chrome and enter chrome://version in the URL bar. In the info screen that will appear, the board type and variant will show up in the Platform and Customization ID fields, respectively:

How to log into the DUT through SSH?¶

Reference: How to SSH to DUT without a password

In some scenarios it could be useful or needed to run certain console commands in the DUT or to retrieve data from it that can only be accessed through a terminal interface. If the DUT is running a test ChromeOS release, it’ll have an SSH daemon running so you can connect to it remotely.

Requirements:

A Linux PC with an SSH client installed. If running Ubuntu, it can be installed with the following command: sudo apt install ssh
The Linux host must be able to ping the DUT. The easiest way to achieve this is to connect the DUT and the Linux PC to the same local network. If the DUT is connected to a Moblab through a wired connection you can also connect to the Linux PC using a wireless connection

Assuming the host you’re connecting from is running Linux, in order to connect to the DUT you’ll need to download the ChromeOS test keys and configure your ssh client properly following these steps:

Download the SSH keys from this link and copy them to ~/.ssh in the Linux host
Set the correct file permissions for the private key:
Get the IP address of the DUT. If the DUT is connected to multiple networks, we need the IP address of the NIC that’s connected to the Linux PC network. To list the available connections and their IP addresses go to virtual terminal 2 in the DUT ([ Ctrl ] [ Alt ] [ → ]) and type: ip -4 -br a

In this example, if the DUT is connected to the Linux PC network through a wireless link, then we can check that the PC can ping the DUT at 192.168.1.137.
Add the following to ~/.ssh/config:

Where $IP_ADDRESS is the IP address of the DUT

How to check the list of peripherals detected by fwupd?¶

If, for debugging purposes, you need to check the current list of peripherals that fwupd is detecting, you can do so by running this command in a DUT terminal:

How to stop the DUT from rebooting automatically¶

If the DUT was provisioned (updated) using Moblab, it will check for an Ethernet link and it will reboot if it doesn’t find one. Make sure to keep the DUT connected to the Moblab network using an Ethernet link.

When you are done testing with the Chromebook, reflash it with a recovery image to prevent it from restarting continuously when not connected to a network through the Ethernet port.

How to send debug information¶

In case something goes wrong when launching or running a test, you can get the complete Moblab logs for debug through the Mobmonitor menu:

Then you can either download the logs or send them to your Cloud Storage bucket:

The logs will be compressed as a .tgz file.

Alternatively, if there’s an issue with Moblab you can report it by issuing a buganizer ticket using this template and filling in the details.