Another day, another Selenium gotcha or two

1. Entering text into rich text editors (iframes with content editable).

Basic:

$this->frame('frame-id');
$this->keys('the text');
$this->frame(null);

Works in Firefox.

In Chrome, you also need:

$this->frame('frame-id');
$this->one('body')->click();
$this->keys('the text');
$this->frame(null);

Note that because we're in the 'scope' of the iframe, $this->one('body') is the body element of the iframe.  Clicking on it gives it the focus so that the keys() get to the iframe.

Doesn't work in IE (10 on Win 7, at least).  Win 7 can't find the body element (although putting in an echo $this->source()) confirms it's finding the iframe OK.

Couldn't work this one out.  The only thing I could do with this is a browser sniff for IE, and if it is inject some javascript to set the value manually.  Because the text editor is empty, I can't click() on any other elements in the iframe.  Horribly hacky.

2. Auto correct

I was putting the text 'i have a comment' into a normal text editor.  Turns out IE autocorrects that to 'I have a comment' which means the test fails when it compares the strings after saving.

Grrr.....

On why writing Selenium tests is like writing raw js 5 years ago

I really thought js libs meant we didn't need to worry quite so much about cross browser quirks so much any more.

Writing some Selenium functional tests, it turns out that's not true.

I have a table. A simple table. With a header, footer, and body content.  With the html in the correct order, as it should be, thead then tfoot then tbody.

In my Selenium tests I generate a lot of tables.  I don't want to compare them cell by cell, so I'd taken the (what I thought to be) neat approach of grabbing the text the first time the test runs, saving it in a file, and then comparing subsequent runs with this known good.

Except.

It turns out that when running the test on Firefox, a call to $element->text() returns the text, without any html, as it appears on screen - that is, thead content, followed by tbody content, followed by tfoot content.

In Chrome, it returns the text, without any html, as the html is - that is, thead content, followed by tfoot, followed by tbody.

For the record, on IE it does the same as Firefox. (This all on a Win7 VM, IE10, Chrome/FF latest).


Which means the tests ALL FAIL when they shouldn't.


Which feels like trying to write cross-browser js all over again.  Painful.


UPDATE 28/4/14

I was wrong.  I jumped the gun.

The tables I was looking at are javascript generated, and inject a tfoot to hold some summary data.  It turns out that Chrome injects the tfoot between the thead and tbody, and so that's what it shows when you get the text content.  Firefox and IE append it to the markup, so it goes after the tbody.  So that's what shows up when you get the text content in FF/IE.

I've yet to look into why the html ends up different in the different browsers; it's possible that's where the bug is.

Anway, what I have learned: don't file bugs on a Friday afternoon.  Make sure your error report actually produces the error.

Set the time date on Virtualbox Windows VM for Selenium testing

When I initialize my Selenium setup before running the tests, two things happen.  First, the database is restored to a known state.

Secondly, the date and time is set to a known, fixed value.

This happens on the Linux VMs that run the application by calling a bash script over ssh on the servers.  That works fine.

However, I also need the Windows VM that's running as a Selenium node to be set at the same time/date.

The reason for this is this.  About the first thing you see when you log in is a diary.  I need that diary to be showing a particular date so that I know what records should be visible, and my tests verify that those records are there.  If I write the tests one week, and run them the next, without resetting the date the diary will be showing the current week - where the data displayed is different.

There are lots of situations in the application where this is the case.  The alternatives (for example, running an sql update on all date columns) seem more wrong to me - my fixtures (the database) are no longer fixed.  I'm pretty sure that trying that will bite me in other ways (for example, the tests will be filled up with $expected = date('d/m/Y'); instead of $expected = '13/12/2013';).

Anyway, there's also some situations where the client uses the local date to calculate things (in particular, there's some javascript that does 'number of days since x'.  So when I run the tests, the date that javascript uses as the starting point to do that calculation needs to be consistent, else the expected value one day is different on the following day.

Now, Windows has a date-time setting to get the date from the internet.  We can turn that off easily in the date settings on the Windows VM.

However, that's only half the story: even with that off, the VM is getting the date/time from the host bios.  So I change the date manually, and a second or two later it switches back to the current date.

I don't want to fix the date on the host machine.  Other bad things will start happening.  For example, all my Jenkins builds will apparently happen at the same time on the same day - the history will be meaningless.  And other things, I'm sure.

So the solution in this situation is obviously to google and try things you don't fully understand until one of them works.  One of these solutions:

host$ vboxmanage setextradata "win7box" "VBoxInternal/Devices/VMMDev/0/Config/GetHostTimeDisabled" "1"

Didn't work.  This is possibly because of Guest Additions (which I need for various reasons).  The next one did.  You need to change the parameters passed to the VirtualBox Guest Additions Service (this is on a Windows VM).  The details are at https://forums/virtualbox.org/viewtopic.php?t=24057#p149903 but you need to add --disable-timesync to the VBoxService.exe call.

Next step is then to run a little batch script to set the date/time when the machine starts.  Obviously this is made more complicated by the fact that you can't change the date as a normal user, you need to be Administrator.  So I can't just bung the .bat file into the startup.

Instead, it seems it needs to be a scheduled task.  To add these, click Start, right-click Computer, click 'Manage'.  Click Task Scheduler on the left, Create Basic Task on the right, follow the wizard.

For reasons that I don't understand, I can't start the selenium node in this way - a shortcut to a batch file to do that stays in the Startup folder.

Adding Windows VM to Selenium grid

Note to self for next time I have to do it:

Import appliance from downloaded images

• Add shared folder (WindowsVM/GuestShare)
• Add VBoxGuestAdditions.iso as storage CD
• Networking - bridged to wlan0

Fire up VM

• Run VBoxGuestAdditions to provide shared folder (restart)
• Install browsers
• Add lamplight to drivers/etc/hosts 
• Copy start-selenium.bat to desktop (so it runs without security warning)
• Add shortcut to Startup folder so it registers with the node on login

Checking pages quickly in Selenium

My first lot of Selenium tests would painstakingly enter the data into a form, save it, and then painstakingly check that the data entered was showing up in the right place, in the right way.

A useful tip from Matthias Verraes at PHPUK conference was to use levenstein string comparison function to just look at the page, and compare it to a known good version.

So here's some helper code in the parent Test class.

First, we run the test and save the results.  We're assuming that everything's working at this point.

    /**
     * @param $content string
     */
    protected function saveResult ($content) {
        $filename = $this->generateFilenameForSavedContent();
        file_put_contents($filename, $content);
        $this->markTestIncomplete("This test is still saving results");
    }

    /**
     * @return string
     */
    private function generateFilenameForSavedContent () {
        $trace = debug_backtrace();
        // We want the name of the test class/method, so we jump up two
        // steps up the backtrace (as this is called by saveResult()
        $caller = array_slice($trace, 2, 1);
        return './expectedresults/' . $caller[0]['class'] . '-' . $caller[0]['function'];
    }

The tests then include a $this->saveResult($this->one('#idOfReturnElement')->text()); in them. This saves the text content (stripped of html, which is fine for me for current purposes) in the file. Note that the naming scheme means there can only be one file per test - that suits me and is probably a good limitation to place on the tests.

They also mark the test as incomplete: once these files are generated, the saveResult() call can and should be removed from the tests. There could, I suppose, be a file_exists() check, and only save and markIncomplete() if it doesn't. But then all the tests will all say ->saveResult() in them, which should be misleading most of the time. Anyway, this is how I've done it and then a quick find/replace removes the saveResult() calls in the tests.

In fact they're replaced by a call to the following method:

    /**
     * @param $actualContent string
     * @param $howCloseInPercent int
     * @return bool
     */
    protected function matchesSavedResult ($actualContent, $howCloseInPercent = 100) {
        $filename = $this->generateFilenameForSavedContent();
        $expectedContent = file_get_contents('./expectedresults/' . $filename);
        
        if ($howCloseInPercent === 100) {
            return ($actualContent === $expectedContent);
        }

        similar_text($actualContent, $expectedContent, $percent);

        return ($percent > $howCloseInPercent);
    }

This doesn't use the levenstein string comparison function: although it performs faster, it has a character limit of 255 which is too low in general. similar_text() is slower but doesn't have a string limit. If we want identity that's easy, but the percent closeness allows us some latitude, depending on the nature of the test. If we're passing in complete html, rather than just the textContent of the DOM, we might want a bit more latitude to allow for css changes, for example.

It all seems to work nicely, and has sped up writing tests no end.  I still have to write the input stuff, but then write the tests once to save the results (and commit them to git), remove the saveResults() calls and replace with matchesSavedResult().

Functional testing emails with mailcatcher, Zend_Mail and more

This is how I've got functional testing of email sending set up.

The idea is that in testing environment emails get caught on the server and phpunit can then query to check that the outgoing email looks correct.

1. Mailcatcher.

Mailcatcher is an SMTP server that receives email and provides a web, and more importantly REST API to see the emails sent.  So step 1 is to install it on the php application server VM.  It could, I suppose, have gone on a separate VM but that seemed overkill.

Install followed these instructions except that I needed some extra bits and bobs (on a Debian 7 box).  The error messages and some trial and error were enough to get this sorted.

sudo apt-get install ruby rubygems

sudo apt-get install ruby1.9.1-dev

sudo apt-get sqlite3 libsqlite3-dev

gem install mailcatcher

and the php.ini setting was actually

sendmail_path = "/usr/bin/env /var/lib/gems/1.9.1/gems/mailcatcher-0.5.12/bin/catchmail "

I don't think that in actual fact this is necessary in my case, because I'm using a Zend Framework SMTP transport, but setting this was helpful for testing while getting it all set up, and probably better safer to do just to avoid accidental spam mail.

Just running 'mailcatcher' works.  But that has it running on localhost (127.0.0.1) which isn't OK because I need to be able to access it from outside the VM.  mailcatcher is on the application server and my phpunit tests are running on the host box.  So starting like this:

$> mailcatcher --http-ip=0.0.0.0

did the job: it's now accessible from the host.  That one-liner is in a little bash script that can be called by the phing task on the host box that does all the VM initialisation prior to running tests (database reset, date/time reset, and now this).

So now I can see the mailcatcher interface at http://lamplight:1080 from the host box (where lamplight is mapping to that VM, of course).

2. Configuring the application.

We're using Zend_Mail (ZF 1, still) and the default transport is configured in the bootstrap.  The transport config options are in the core config file, so all I needed to do was set the host and port to 127.0.0.1 and 1025 respectively (I have a [functest] section of the config file).  It actually took me ages to get this working, but in the end it was irritatingly simple.

3. Querying mailcatcher from tests

I'm using a mailcatcher client without the behat stuff, which was as easy as adding "alexandresalome/mailcatcher": "*" to my composer.json file in the testing folder.  I can now use that client class in my phpunit tests to check that the emails sent are correct.

4. And next

I wonder if it'd be useful to have some assertions along the lines of this:

    $email = $this->retrieveLastEmailSent();
    $this->assertEmailRecipients(["bob@example.com", $email);
    $this->assertEmailSubject("Testing my lovely emails", $email);
    $this->assertEmailBody("Hi Bob, this is a test email", $email);

    $email2 = $this->retrieveEmailSent(1);  // gets the last but 1
    $this->assertEmailRecipients(["tracey@example.com", $email2);
    $this->assertEmailBody("Hi Tracey, this is a test email", $email2, "See how email-merges work!");
    $this->assertEmailAttachments("md5ofexpectedfileorsomething?", $email2);
    $this->assertEmailFrom("matt@example.com", $email2);



Along the lines of http://codeception.com/12-15-2013/testing-emails-in-php.html, most likely, although I think I'd like to have a try at creating my own assertion classes 'properly' (i.e. the way it's suggested in the phpunit manual in the section on extending it).

And a big thanks to the lovely people that have made these tools (Samuel Cochran and Alexandre Salomé).

Adding CLI options to PHPUnit

Issue:  I've got PHPUnit set up to run Selenium tests on varying combinations of browsers/platforms.  When I'm writing tests and doing day-to-day stuff, I want to run just on one (a VM with Firefox on Win 7, at the moment).  When I want to look into a new IE11 bug, I want to run a particular test just on Win 8/ IE 11.  When I'm running a full test run, I want to run across all target browsers/platforms.

The browser/platform combination to use has been specified up to now as a static property of my base TestCase class.  It's a shorthand: I can set it to 'FF' for Firefox on Win7, 'IE' for Internet Explorer 10 on Win7, '8' for Firefox, IE11 and Chrome on Windows 8, or 'all' for 3 browsers across Win7 and Win8.

It's not much, but I was wanting to set this as a command line option, rather than changing the value of the property on the class manually:

$ phpunit --browser-set="FF" tests/GoTestSomeStuffTest.php

The phpunit code makes it sound like this'll be straightforward.  Just subclass PHPUnit_TextUI_Command, add in a hook to the constructor and handler... easy.

Almost.

The suggested code is:

     * <?php

     * class MyCommand extends PHPUnit_TextUI_Command

     * {

     *     public function __construct()

     *     {

     *         $this->longOptions['--my-switch'] = 'myHandler';

     *     }

     *

     *     // --my-switch foo -> myHandler('foo')

     *     protected function myHandler($value)

     *     {

     *     }

     * }

(https://github.com/sebastianbergmann/phpunit/blob/master/src/TextUI/Command.php#L210)

Firstly, that's not quite right.  Looking at the other longOptions, it's clear that the initial double hyphen shouldn't be there:

     *     public function __construct()

     *     {

     *         $this->longOptions['my-switch'] = 'myHandler';

     *     }

Secondly, if you want to pass an option value you need to append an equals sign to the switch name:

     *     public function __construct()

     *     {

     *         $this->longOptions['my-switch='] = 'myHandler';

     *     }

But how do we get to use MyCommand instead of PHPUnit_TextUI_Command?  I'm not sure if this is the right answer, but it's an answer.  I've copied the bash script phpunit that starts it all up (from https://github.com/sebastianbergmann/phpunit/blob/master/phpunit), and instead of 

PHPUnit_TextUI_Command::main();

I have

require_once 'MyCommand.php';

MyCommand::main();

(OK it's not really called MyCommand, I'm just staying with the original docs!).  Plus a chmod to 755 to get it to run.

Almost there.  However, our new MyCommand constructor isn't being called still.  main() on the original looks like this:

    /**

     * @param boolean $exit

     */

    public static function main($exit = true)

    {

        $command = new static;



        return $command->run($_SERVER['argv'], $exit);

    }

so the line new static  gives us an instance of PHPUnit_TextUI_Command, because that's where we are when it gets called.  So MyCommand needs this method too, just copied straight across.  In the end it looks like this:

     <?php

     class MyCommand extends PHPUnit_TextUI_Command

     {

         public function __construct()

         {

             $this->longOptions['browser-set='] = 'setBrowserSet';

         }

     

         // --browser-set="FF" -> setBrowserSet("FF")

         protected function setBrowserSet($value)

         {

             MyTestSetup::setBrowserSetToUse($value);

         }




         public static function main($exit = true) 

         {

             $command = new static;

             return $command->run($_SERVER['argv'], $exit);

         }

     }

Validation and a default value, and handling it all, sit in MyTestSetup (which indirectly extends PHPUnit_Extensions_Selenium2TestCase).

Note there is a way to define $browsers in a configuration somehow - I think through phpunit.xml (I seem to remember seeing it there) but it sets it on PHPUnit_Extensions_SeleniumTestCase, but I'm using PHPUnit_Extensions_Selenium2TestCase and so don't think it'll work.    

(Edit: excuse the horrible code line spacing.  Another to-do for the list.)