about summary refs log tree commit diff
path: root/nixos/tests/chromium.nix
AgeCommit message (Collapse)AuthorFilesLines
2022-11-06nixos/tests/chromium: Re-enable the chrome://gpu test for M107Michael Weiss1-5/+6
I need to fix copying the chrome://gpu content to the clipboard (Ctrl+a doesn't work anymore so we have to click the button) but we can at least test the font rendering for now.
2022-10-24nixos/tests/chromium: Disable a failing test for M107+Michael Weiss1-4/+8
I haven't had time to look into this yet but it looks like opening chrome://gpu doesn't work anymore without proper GPU rendering (we use software rendering due to the virtualisation). According to the console output the new window never opens (at least with Google Chrome - I couldn't test it with Chromium yet due to the failing builds for M107 and M108): ``` (finished: sending keys ‘chrome://gpu ‘, in 0.14 seconds) machine: waiting for a window to appear machine: must succeed: xwininfo -root -tree | sed 's/.*0x[0-9a-f]* \"\([^\"]*\)\".*/\1/; t; d' (finished: must succeed: xwininfo -root -tree | sed 's/.*0x[0-9a-f]* \"\([^\"]*\)\".*/\1/; t; d', in 0.05 seconds) machine # Error: eglChooseConfig returned zero configs machine # at Create (../../third_party/dawn/src/dawn/native/opengl/ContextEGL.cpp:53) machine # machine: must succeed: xwininfo -root -tree | sed 's/.*0x[0-9a-f]* \"\([^\"]*\)\".*/\1/; t; d' machine # WARNING: lavapipe is not a conformant vulkan implementation, testing use only. (finished: must succeed: xwininfo -root -tree | sed 's/.*0x[0-9a-f]* \"\([^\"]*\)\".*/\1/; t; d', in 0.06 seconds) machine: must succeed: xwininfo -root -tree | sed 's/.*0x[0-9a-f]* \"\([^\"]*\)\".*/\1/; t; d' (finished: must succeed: xwininfo -root -tree | sed 's/.*0x[0-9a-f]* \"\([^\"]*\)\".*/\1/; t; d', in 0.09 seconds) [...] ```
2022-10-24nixos/tests/chromium: Fix the tests for Google ChromeMichael Weiss1-0/+2
The meta attribute "timeout" is only set for Chromium (might still be required due to the long build duration). The Google Chrome tests were failing with: error: attribute 'timeout' missing According to nixos/lib/testing/meta.nix "null values are filtered out by `meta`" so `timeout = chromiumPkg.meta.timeout or null` might be fine as well.
2022-05-14nixos/tests/chromium: Switch to nodes.machine (vs. deprecated machine attribute)Michael Weiss1-6/+8
The old attribute is deprecated: trace: warning: In test `chromium-stable': The `machine' attribute in NixOS tests (pkgs.nixosTest / make-test-pyton.nix / testing-python.nix / makeTest) is deprecated. Please use the equivalent `nodes.machine'. Note: This is only a refactoring.
2022-03-18nixos/tests: fix some evaluation errorsNaïm Favier1-18/+20
Fixes errors caught by "nixos/testing: restrict arguments to makeTest" as well as some unrelated errors and warnings.
2021-11-06nixosTests.chromium: Avoid blocking on xclip stdoutRobert Hensing1-1/+1
2021-11-05nixosTests.chromium: Redirect stdout to avoid blockingRobert Hensing1-1/+1
2021-11-01nixos/tests/chromium: Add a workaround to fix the test for M96+Michael Weiss1-0/+4
Some upstream changes broke the automatic fallback to SwiftShader. Without this workaround the GPU initialization fails (apparently due to requiring Vulkan): machine # libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null) machine # [1001:1001:1101/104304.000625:ERROR:viz_main_impl.cc(161)] Exiting GPU process due to errors during initialization machine # libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null) machine # [1052:1052:1101/104305.060496:ERROR:viz_main_impl.cc(161)] Exiting GPU process due to errors during initialization machine # [1084:1084:1101/104305.165898:ERROR:angle_platform_impl.cc(44)] Display.cpp:894 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-3): Initialization of an object could not be completed for implementation-specific reasons, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, initialize:1048. machine # [1084:1084:1101/104305.171191:ERROR:gl_surface_egl.cc(782)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-3): Initialization of an object could not be completed for implementation-specific reasons, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, initialize:1048. machine # [1084:1084:1101/104305.178707:ERROR:gl_surface_egl.cc(1382)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED machine # [1084:1084:1101/104305.180111:ERROR:gl_ozone_egl.cc(20)] GLSurfaceEGL::InitializeOneOff failed. machine # [1084:1084:1101/104305.189760:ERROR:viz_main_impl.cc(161)] Exiting GPU process due to errors during initialization machine # [1092:1092:1101/104305.233470:ERROR:gpu_init.cc(457)] Passthrough is not supported, GL is disabled, ANGLE is This workaround restores the previous result: machine # [1004:1004:1101/104551.131190:ERROR:gpu_init.cc(457)] Passthrough is not supported, GL is swiftshader, ANGLE is The workaround is only required for Chromium, not Google Chrome.
2021-07-25nixos/tests/chromium: Drop the workaround for Chrome GPU crashesMichael Weiss1-5/+1
This regression was fixed by 51d83077ffb.
2021-07-25nixos/tests/chromium: Check the version and that it's an official buildMichael Weiss1-0/+10
This also prints and screenshots the output of chrome://version which contains useful information. Outputs (stable, beta, ungoogled, chrome-stable, chrome-beta, chrome-dev): Chromium 92.0.4515.107 (Official Build) (64-bit) Chromium 92.0.4515.107 (Official Build) (64-bit) Chromium 91.0.4472.164 (Official Build, ungoogled-chromium) (64-bit) Google Chrome 92.0.4515.107 (Official Build) (64-bit) Google Chrome 92.0.4515.107 (Official Build) beta (64-bit) Google Chrome 93.0.4577.8 (Official Build) dev (64-bit)
2021-07-23chromium: Check the text renderingMichael Weiss1-1/+2
This should catch regressions like #131074 in the future. In that case a glibc update caused a regression that caused most of the text to become invisible (just not the "Web Store" we've already been checking for).
2021-07-21chromium: 91.0.4472.164 -> 92.0.4515.107Michael Weiss1-1/+1
https://chromereleases.googleblog.com/2021/07/stable-channel-update-for-desktop_20.html This update includes 35 security fixes. CVEs: CVE-2021-30565 CVE-2021-30566 CVE-2021-30567 CVE-2021-30568 CVE-2021-30569 CVE-2021-30571 CVE-2021-30572 CVE-2021-30573 CVE-2021-30574 CVE-2021-30575 CVE-2021-30576 CVE-2021-30577 CVE-2021-30578 CVE-2021-30579 CVE-2021-30580 CVE-2021-30581 CVE-2021-30582 CVE-2021-30583 CVE-2021-30584 CVE-2021-30585 CVE-2021-30586 CVE-2021-30587 CVE-2021-30588 CVE-2021-30589 Note: This won't be the smoothest update. Chromium seems to be fine but requires gtk3 in $LD_LIBRARY_PATH to find libgtk-3.so.0 (otherwise it crashes during startup) but Google Chrome fails to initialize ("GPU process exited unexpectedly: exit_code=132") and requires "--use-gl=angle --use-angle=swiftshader" for hardware(?) acceleration (which seems to work work fine and performant but SwiftShader should actually use the CPU instead of the GPU).
2021-07-10nixos/tests/chromium: Print the content of chrome://{sandbox,gpu}Michael Weiss1-46/+27
This can be very useful when running the test headless or e.g. when looking at Hydra logs. Especially the chrome://gpu content contains a lot of interesting information. I also decided to refactor the test_new_win() function to avoid duplicate code and rely less on xdo.
2021-07-10nixos/tests/chromium: Refactor launching the browser processMichael Weiss1-13/+13
It should now be more flexible and less error-prone.
2021-07-09nixos/tests/chromium: Fix the test for M92+Michael Weiss1-11/+19
Unfortunately there are some regressions in the GPU code that cause Chromium and Google Chrome to crash, e.g.: machine # [0709/084047.890436:ERROR:process_memory_range.cc(75)] read out of range[ 30.153484] show_signal: 20 callbacks suppressed machine # [ 30.153490] traps: chrome[1036] trap invalid opcode ip:55af03357b29 sp:7ffeaa69ad10 error:0 in chrome[55aefe7a4000+81ec000] machine # machine # [0709/084047.955039:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2) machine # [0709/084047.955078:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: No such file or directory (2) machine # [ 30.126905] systemd[1]: Created slice system-systemd\x2dcoredump.slice. machine # [ 30.137012] systemd[1]: Started Process Core Dump (PID 1038/UID 0). machine # [ 30.571987] systemd-coredump[1039]: Process 1036 (chrome) of user 1000 dumped core. machine # [992:1021:0709/084048.501937:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=132 machine # [ 30.594747] systemd[1]: systemd-coredump@0-1038-0.service: Succeeded. Hopefully this'll be fixed upstream before the final release (there are bug reports for it) but for the meantime we have to launch the beta and dev versions with "--use-gl=angle --use-angle=swiftshader".
2021-05-09nixosTests.chromium: lintMaximilian Bosch1-1/+2
Note: I didn't execute it entirely because I'd have to build chromium for this, but the diff appears fine.
2021-01-24nixos/tests/chromium: Simplify the logic (#110715)Michael Weiss1-75/+40
- Improve the documentation (Python docstrings for functions and more meaningful xdotool script names). - Make more use of the existing methods (e.g. wait_until_succeeds(), and send_key()) - Note: This increases timeouts from 60 seconds to (currently) 15 minutes but the defaults from test-driver.py should be fine. This should make it simpler to read the code, understand the test output, and drop the custom xdotool scripts in the future.
2021-01-23nixos/tests/chromium: Fix another potential race conditionMichael Weiss1-9/+10
The create_new_win() function could open multiple windows when used incorrectly. This change makes sure that a new window will only be created if the main window could be selected successfully. This also ignores the out return values as they're never used.
2021-01-20nixos/tests/chromium: Add ungoogled-chromium and Google ChromeMichael Weiss1-6/+30
By default only `chromium` will be tested but other "channels" can be selected using e.g.: nix-build nixos/tests/chromium.nix -A ungoogled This also adds me as secondary maintainer (I'd like to get notified on PRs/issues and can review them).
2021-01-20nixos/tests/chromium: Fix a race conditionMichael Weiss1-7/+8
Only execute Ctrl+w to close the currently active window if the new/secondary window (title: "New Tab") could be selected. This fixes a test failure since the update to Chromium M88 (cc PR #110010). Without this additional check the main window (title: "startup done") could still be selected (and thus will be closed) and the script would close both windows (i.e. terminate Chromium completely).
2020-05-03treewide: use https for nixos.org and hydra.nixos.orgPavol Rusnak1-1/+1
tarballs.nixos.org is omitted from the change because urls from there are always hashed and checked
2020-02-06nixosTests.chromium: Port to PythonJacek Galowicz1-151/+216
2020-01-29nixos/display-managers/auto: removeworldofpeace1-1/+1
This module allows root autoLogin, so we would break that for users, but they shouldn't be using it anyways. This gives the impression like auto is some special display manager, when it's just lightdm and special pam rules to allow root autoLogin. It was created for NixOS's testing so I believe this is where it belongs.
2020-01-07nixos/tests/chromium: Fix nixos-hex.svg hashRickard Nilsson1-1/+1
2018-11-11tests: refactor to carry the package set as an argumentLéo Gaspard1-2/+3
This way, the package set will be possible to pass without re-importing all the time
2018-11-02nixos/tests/chromium: Fix sandbox info matchingaszlig1-2/+2
As reported by @andir, the regular expressions that match the sandbox output are no longer matching in the recent Chromium bump as of bb03fbc2c8b97ef5a4f4205eee51029155164b8a. Instead of a boolean field that determines whether namespace sandboxes are on, the namespace sandbox is now an enum within "Layer 1 Sandbox". I've modified the regular expressions accordingly and also ran the test for the stable branch, which now succeeds. Signed-off-by: aszlig <aszlig@nix.build> Issue: https://github.com/NixOS/nixpkgs/issues/49442 Cc: @bendlas, @andir
2018-11-01chromium tests: inherit timeout from the packageVladimír Čunát1-2/+4
/cc #49442. It should decrease the waste of resources due to abortions.
2018-04-28nixos/tests/chromium: Wait 10s after new windowaszlig1-0/+5
This is a very very very ugly workaround and it's because Chromium seems to eat keystroke for a few seconds after a new window is created. I haven't found a better solution yet, so let's at least unbreak the test until we come up with a better way. Thanks to @vcunat for bringing this to my attention and also doing the initial bisect. The change that brought up this problem was 2b29e401531306d044f797a5dfa, which updated Chromium from version 65.0.3325.181 to version 66.0.3359.117. Unfortunately the upstream changelog[1] is way too large to actually guess what the breaking change is. [1]: https://chromium.googlesource.com/chromium/src/+log/65.0.3325.181..66.0.3359.117?pretty=fuller&n=10000 Signed-off-by: aszlig <aszlig@nix.build> Cc: @bendlas, @vcunat
2018-04-09nixos/tests.chromium: actually notice the tab crashMichael Raskin1-2/+22
Argh, debugging NixOS tests takes forever… (cherry picked from commit 1afb6e790a6a28a4fc5f310a2b0e5e012eb416fd)
2017-02-07nixos/tests/chromium: Run tests as normal useraszlig1-18/+21
The tests have failed because Chromium has started up displaying the following error message in a dialog window: Chromium can not be run as root. Please start Chromium as a normal user. If you need to run as root for development, rerun with the --no-sandbox flag. So let's run as user "alice" and pass all commands using the small helper function "ru" (to keep it short, it's for "Run as User"). Tested it by running the "stable" test on x86_64-linux. Signed-off-by: aszlig <aszlig@redmoonstudios.org> Reported-by: @globin
2016-11-09nixos/tests/chromium: Fix popup detectionaszlig1-1/+1
Sometimes it happens that the "Type to search or enter a URL to navigate" popup doesn't show, but all we need to know at this time is whether Chromium has finished starting up. So checking for the "startup done" page is a better option here. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-03-21nixos/tests/chromium: Propagate "system" to pkgsaszlig1-1/+1
Assigning the channelMap by the function attrset argument at the top-level of the test expression file may reference a different architecture than we need for the tests. So if we get the pkgs attribute by auto-calling, this will lead to test failure because we have a different architecture for the test than for the browser. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-03-20nixos/tests/chromium: Allow overriding channel mapaszlig1-6/+9
This has been the case before e45c211, but it turns out that it's very useful to override the channel packages so we can run tests with different Chromium build options. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-03-01nixos/tests/chromium: Split up into subtestsaszlig1-73/+62
This makes it easier to test just a specific channel rather than to force testing all builds down the users/testers throat. Especially this makes it easier to test NixOS channel upgrades only against the Chromium stable channel instead of just removing the beta/dev channels from the tests entirely (as done in 69ec09f38aa1f1d37baec73ebdf9cf5f21050f94). Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-02-03Don't make chromium-beta/dev release blockersEelco Dolstra1-2/+2
Generally we shouldn't ship pre-release versions anyway, and we certainly don't want them to be release blockers. Also, chromium builds are just too slow to have them blocking the channel (see https://github.com/NixOS/nixpkgs/issues/12794).
2015-11-29tests-chromium: fix link to svg filePascal Wittmann1-2/+2
closes #11208
2015-09-12xdotool: 2.20110530.1 -> 3.20150503.1Gabriel Ebner1-0/+6
2015-08-25GrmblEelco Dolstra1-1/+1
http://hydra.nixos.org/build/24983009
2015-08-24chromium: Fix test on i686-linuxEelco Dolstra1-1/+1
http://hydra.nixos.org/build/24982682
2015-08-20Give the chromium test more memoryEelco Dolstra1-1/+1
Hopefully fixes random failures like http://hydra.nixos.org/build/24861987/nixlog/446
2015-07-12all tests: added meta.maintainers sectionJoachim Schiele1-0/+3
2015-07-04nixos/tests/chromium: Improve sandbox checking.aszlig1-2/+3
We no longer need have "SUID sandbox" enabled in the chrome://sandbox status page and we now also check for "You are adequately sandboxed." to be absolutely sure that we're running with proper sandboxing. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-05-22nixos/tests/chromium: Detect popup using OCR.aszlig1-5/+3
This will make the test a lot more reliable, because we no longer need to press ESC multiple times hoping that it will close the popup. Unfortunately in order to run this test I needed to locally revert the gyp update from a305e6855dd8723683c77635f45ae28411c8f36c. With the old gyp version however the test runs fine and it's able to properly detect the popup. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-04-02tests/chromium: Work around popup close flakiness.aszlig1-1/+6
It's not nice to send the escape key over and over again just to ensure the popup is closed, because even *if* it fails to close the popup 4 times in a row it's just very unlikely that it will be closed. But in order to make really sure, we might need to do a screenshot and detect visual changes. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-02-22nixos/tests/chromium: Check new userns sandbox.aszlig1-1/+1
Since Chromium version 42, we have a new user namespaces sandbox in the upstream project. It's more integrated so the chrome://sandbox page reports it as "Namespace Sandbox" instead of SUID sandbox, which we were re-using (or abusing?) in our patch. So if either "SUID Sandbox" or "Namespace Sandbox" reports with "Yes", it's fine on our side. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-01-26nixos/tests/chromium: Increase VM memory size.aszlig1-0/+1
Chromium is quite memory hungry and we frequently get random crashes in the tests, so let's set it to 1024 MB because new releases of Chromium most probably won't consume *less* memory. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-11-19nixos/tests/chromium: Allow to override packages.aszlig1-6/+13
Of course, this could be done via packageOverrides, but this is more explicit and makes it possible to run the tests with various Chromium overrides. Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2014-11-18nixos: Add rudimentary VM tests for Chromium.aszlig1-0/+157
Currently, the test is only for testing the user namespace sandbox and even that isn't very representative, because we're running the tests as root. But apart from that, we should have functionality for opening/closing windows and the main goal here is to get them as deterministic as possible, because Chromium usually isn't very nice to chained xdotool keystrokes. And of course, the most important "test" we have here: We know at least whether Chromium works _at_all_. Signed-off-by: aszlig <aszlig@redmoonstudios.org>