about summary refs log tree commit diff
path: root/nixos/lib/test-driver
AgeCommit message (Collapse)AuthorFilesLines
2021-06-13nixos/tests/test-driver: cleanup nix expressionDavid Arnold1-1/+1
Less nesting, where that improves readability. More nesteing, where that improves readability, but most importantly: Expose individual functions separately so that they can be more easily built directly, eg.: `nix build --impure --expr '(import ./testing-python.nix {system = builtins.currentSystem;}).mkTestDriver'`
2021-06-06nixos/tests/test-driver: cleanup "dead" code (USE_SERIAL)David Arnold1-1/+0
At nixpkgs root: `rg redirectSerial ./` does not result in any other match nor does `rg USE_SERIAL ./` except for an unrelated match in: pkgs/tools/graphics/argyllcms/default.nix
2021-06-05nixos/test-driver: Run commands with error handlingtalyz1-1/+1
Bash's standard behavior of not propagating non-zero exit codes through a pipeline is unexpected and almost universally unwanted. Default to setting `pipefail` for the command being run; it can still be turned off by prefixing the pipeline with `set +o pipefail` if needed. Also, set `errexit` and `nonunset` options to make the first command of consecutive commands separated by `;` fail, and disallow dereferencing unset variables respectively.
2021-06-03Merge pull request #125372 from Synthetica9/shell_interactDomen Kožar1-0/+11
nixos/tests/test-driver: add shell_interact
2021-06-03nixos/tests/test-driver: make it clear when shell is readyPatrick Hilhorst1-0/+1
Co-authored-by: Domen Kožar <domen@enlambda.com>
2021-06-02nixos/tests/test-driver: document shell_interactPatrick Hilhorst1-1/+1
2021-06-02nixos/tests/test-driver: add shell_interactPatrick Hilhorst1-0/+10
2021-05-30nixos/test-driver: mention the elapsed time when it times outAndreas Rammhold1-3/+3
For now you had to know that the actions are retried for 900s when seeing an error like > Traceback (most recent call last): > File "/nix/store/dbvmxk60sv87xsxm7kwzzjm7a4fhgy6y-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 927, in run_tests > exec(tests, globals()) > File "<string>", line 1, in <module> > File "<string>", line 31, in <module> > File "/nix/store/dbvmxk60sv87xsxm7kwzzjm7a4fhgy6y-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 565, in wait_for_file > retry(check_file) > File "/nix/store/dbvmxk60sv87xsxm7kwzzjm7a4fhgy6y-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 142, in retry > raise Exception("action timed out") > Exception: action timed out in your (hydra) build failure. Due to the absence of timestamps you were left guessing if the machine was just slow, someone passed a low timeout value (which they couldn't until now) or whatever might have happened. By making this error a bit more descriptive (by including the elapsed time) these hopefully become more useful.
2021-05-21Merge pull request #123823 from misuzu/test-driver-usb-boot-speedupJacek Galowicz1-1/+2
nixos/test-driver: use usb-ehci controller instead of piix3-usb-uhci
2021-05-20nixos/test-driver: use usb-ehci controller instead of piix3-usb-uhcimisuzu1-1/+2
On my system this change offers ~5X speed up of nixosTests.boot.biosUsb and nixosTests.boot.uefiUsb tests.
2021-05-14nixos/testing: add interactive serial stdout logs switch and dim themDavid Arnold1-6/+27
2021-05-04nixos/test-driver: Allow interactive testing on Wayland-only setupsMichael Weiss1-1/+2
On my system I have XWayland disabled and therefore only WAYLAND_DISPLAY is set. This ensures that the graphical output will still be enabled on such setups (both Wayland and X11 are supported by the viewer).
2021-04-23nixos/test-driver: use a variety of different Tesseract settings for OCRLuke Granger-Brown1-31/+48
When performing OCR, some of the Tesseract settings perform better than others on a variety of different workloads, but they mostly take ~negligible incremental time to run compared to the overhead of running the ImageMagick filters. After this commit, we try using all three of the current Tesseract models (classic, LSTM, and classic+LSTM) to generate output text. This fixes chromium-90's tests at release-20.09, and should make cases where you're looking for *specific* text better, with the tradeoff of running Tesseract multiple times. To make it sensible to cherrypick this into release-20.09, this doesn't change the existing API surface for the test driver. In particular, get_screen_text continues to have the existing behaviour.
2020-11-21nixos test-driver: fix single line docstrings, fixes #104467Frederik Rietdijk1-4/+2
Single line docstrings should have the """ on a single line according to PEP 8. It seems support for this landed in the latest version of Black.
2020-10-25test-driver.py: remove bufsize=1 from Popen callsKonrad Borowski1-2/+0
According to Python documentation [0], `bufsize=1` is only meaningful in text mode. As we don't pass in an argument called `universal_newlines`, `encoding`, `errors` or `text` the file objects aren't opened in text mode, which means the argument is ignored with a warning in Python 3.8. line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used This commit removes this warning that appared when using interactive test driver built with `-A driver`. This is done by removing `bufsize=1` from Popen calls. The default parameter when unspecified for `bufsize` is `-1` which according to the documentation will be interpreted as `io.DEFAULT_BUFFER_SIZE`. As mentioned by a warning, Python already uses default buffer size when providing `buffering=1` parameter for file objects not opened in text mode. [0]: https://docs.python.org/3/library/subprocess.html#subprocess.Popen
2020-09-11test-driver.py: defaulting keepVmState in Machine initFélix Baylac-Jacqué1-1/+1
ecb73fd5557d6d438aa7c155e5b1aad89373c6ae introduced a new keepVmState CLI flag for test-driver.py. This CLI flags gets forwarded to the Machine class through create_machine. It created a regression for the boot tests where __main__ end up not being evaluated. See https://github.com/NixOS/nixpkgs/pull/97346#issuecomment-690951837 for bug report. Defaulting keepVmState to false when __main__ ends up not being evaluated.
2020-09-07test-driver.py: fix VM state directory deletionFélix Baylac-Jacqué1-9/+12
The previous version of the code would only kick in if the state directory path pointed at a *file*, which never occurs. Making that codepath actually work reveals an ordering bug, which this patch fixes as well. It also replaces the confusing, imperative case log message "delete VM state directory" with "deleting VM state directory". Finally, we hint the user about how to prevent this deletion. IE. by passing the --keep-vm-state flag. Bug report: https://github.com/NixOS/nixpkgs/pull/91046#issuecomment-685568750 Credit goes to Edef for the rebase on top of a recent nixpkgs commit and for writing most of this commit message. Co-authored-by: edef <edef@edef.eu>
2020-08-30Revert "Merge pull request #96254 from Mic92/logging"Anders Kaseorg1-144/+221
This reverts commit 4fc708567f6d9cf28f9ba426702069aa5a0b89c2, reversing changes made to 0e54f3a6d8393c31cfae43316904375dcfc77a46. Fixes #96699. Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-30Revert "Merge pull request #96152 from JJJollyjim/colour-test-machines-staging"Anders Kaseorg1-13/+2
This reverts commit 1bff6fe17cbf3e81fbd4122af41d77ea378f45d7, reversing changes made to 2995fa48cb4878756b9d64b27535737278d96f07. There’s presumably nothing wrong with this PR, except that it conflicts with reverting #96254 which broke several tests (#96699). Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-29nixos/test-driver: Use guest time when using sleepaszlig1-1/+2
With the Perl driver, machine.sleep(N) was doing a sleep on the guest machine instead of the host machine. The new Python test driver however uses time.sleep(), which instead sleeps on the host. While this shouldn't make a difference most of the time, it *does* however make a huge difference if the test machine is loaded and you're sleeping for a minimum duration of eg. an animation. I stumbled on this while porting most of all my tests to the new Python test driver and particularily my video game tests failed on a fairly loaded machine, whereas they don't with the Perl test driver. Switching the sleep() method to sleep on the guest instead of the host fixes this. Signed-off-by: aszlig <aszlig@nix.build>
2020-08-27nixos/lib/test*: remove perl test driverFlorian Klink3-1000/+0
This has been deprecated in 20.03, and all tests have been migrated to the python framework, effectively making this dead code.
2020-08-27nixos/test: colour machine namesJamie McClymont1-2/+13
2020-08-25nixos/test-driver: re-introduce log()Jörg Thalheim1-0/+3
Appearantly this is used in tests
2020-08-25nixos/testdriver: sort importsJörg Thalheim1-7/+8
2020-08-25nixos/test-driver: switch to pythons' logging libJörg Thalheim1-214/+127
- Less code - more thread-safe according to @flokli
2020-08-25nixos/test-driver: introduce main methodJörg Thalheim1-2/+8
This way we not accidentally use introduce/use global variables. Also it explictly mark the code for the mypy type checker.
2020-08-21nixos/testing: Fix fail() functionJanne Heß1-2/+5
The docs say this behaves as succeed(), but it does not return stdout as succeed() does. This fixes that behaviour
2020-08-04nixos/lib/*: editorconfig fixeszowoq1-4/+4
2020-07-06nixos/test-driver: print a traceback when testScript fails (#92369)Justinas Stankevičius1-1/+3
* print a traceback: assertion message can be empty * change print back to eprint
2020-07-01Merge pull request #65231 from buckley310/grub-passwordMichele Guerini Rocco1-0/+23
grub: add support for passwords
2020-06-24nixos/lib/test-driver: add wait_for_console_textrnhmjoj1-0/+23
This method is similar to wait_for_text but is based on matching serial console lines instead of the VGA output.
2020-06-21test-driver.py: delete VM state directory after test runFélix Baylac-Jacqué1-2/+18
Keeping the VM state test across several run sometimes lead to subtle and hard to spot errors in practice. We delete the VM state which contains (among other things) the qcow volume. We also introduce a -K (--keep-vm-state) flag making VM state to persist after the test run. This flag makes test-driver.py to match its previous behaviour.
2020-06-01Merge pull request #82258 from erikarvstedt/fix-xchg-cachingFlorian Klink1-6/+0
fix inconsistent caching of VM xchg dirs
2020-06-01qemu-vm: fix inconsistent caching of xchg dirsErik Arvstedt1-1/+0
xchg is advertised as a bidirectional exchange dir, but file content transfer from host to VM fails due to caching: If a file is read in the VM and then modified on the host, subsequent re-reads in the VM can yield old, cached data. This is caused by the use of 9p's cache=loose mode that is explicitly meant for read-only mounts. 9p doesn't provide any suitable cache modes, so fix this by disabling caching. Also, remove a now unnecessary sync in the test driver.
2020-05-27test-driver: remove useless syncsErik Arvstedt1-5/+0
These syncs have the goal to transfer host filesystem changes to the VM, but they have no effect because 1) syncing in the VM can't possibly pull in host data and 2) 9p is accessing the host filesystem on the cached layer anyways, so even syncing on the host would have no effect in the VM.
2020-05-09nixos/test-driver: Specify /bin/sh shell when running a bourne shell script ↵Chuck1-1/+1
as the user The test harness provides the commands it wishes to run in Bourne syntax. This fails if the user uses a different shell. For example, with fish: machine.wait_for_unit("graphical-session.target", "alice") machine # fish: Unsupported use of '='. To run '-u`' with a modified environment, please use 'env XDG_RUNTIME_DIR=/run/user/`id -u`…' machine # XDG_RUNTIME_DIR=/run/user/`id -u` systemctl --user --no-pager show "graphical-session.target" machine # ^ machine # [ 16.329957] su[1077]: pam_unix(su:session): session closed for user alice error: retrieving systemctl info for unit "graphical-session.target" under user "alice" failed with exit code 127
2020-05-07testing{-python}.nix: Remove log pretty-printing cruftEelco Dolstra3-294/+0
This completes the removal of the nested log feature, which previously got removed from Nix, Hydra, stdenv and GNU Make. In particular, this means that the output of VM builds no longer contains a copy of jQuery.
2020-05-07test-driver.py: Fix deadlock when the log queue gets fullEelco Dolstra1-3/+3
If a program (e.g. nixos-install) writes more than 1000 lines to stderr during execute(), then process_serial_output() deadlocks waiting for the queue to be processed. So use an unbounded queue instead. We should probably get rid of the structured log output (log.xml), since then we don't need the log queue anymore.
2020-05-01nixosTests: drop nr_tests and failed_tests variablesFlorian Klink1-20/+0
With the tests now bailing out early on a failing subtest, we don't need to keep a list of failed tests, or the number of total tests
2020-05-01nixosTests: Reraise exception in subtestsJacek Galowicz1-4/+2
2020-04-06nixos/lib/test-driver: Fix require_unit_state hardcoded formattingSilvan Mosberger1-1/+1
2020-03-29test-driver.py: use temporary dir for vde1.ctlMartin Milata1-3/+5
Send SIGTERM instead of SIGKILL to vde_switch to give it chance to delete the directories.
2020-03-27nixos/test: update test-driver.py for mypy 0.770Rouven Czerwinski1-1/+4
2020-02-14nixosTests: Make VLAN argument optional in python test driverJacek Galowicz1-1/+1
2020-02-07nixos/test: Add copy_from_host and copy_from_host_via_shell to pythonJacek Galowicz1-4/+40
test driver
2020-01-26nixosTests: Drop invalid utf characters in command outputJacek Galowicz1-1/+1
2020-01-19Merge pull request #77587 from singron/fix-sharedFlorian Klink1-2/+2
test-driver.py: Share the shared dir between VMs
2020-01-19test-driver.py: Use /tmp/shared for copy_from_vmEric Culp1-1/+1
The docstring says it uses a directory shared among all vms, although that doesn't seem necessary for the functionality. However, it does need to be consistent between the guest and host.
2020-01-17Merge pull request #77662 from marijanp/verbose-python-test-driverFlorian Klink1-5/+13
nixos/test: added verbose output for failed tests
2020-01-14test-driver.py: fix decoding of VM outputErik Arvstedt1-1/+2
The codec format 'unicode_escape' was introduced in 52ee102 to handle undecodable bytes in boot menus. This made the problem worse as unicode chars outside of iso-8859-1 produce garbled output and valid utf-8 strings (such as "\x" ) trigger decoding errors. Fix this by using the default 'utf-8' codec and by explicitly ignoring decoding errors.
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-08-05 09:40:21 +0000