Headless CHROME With Selenium Web Driver

Till Google Chrome 59, headless execution has been achieved using third party tools like PhantomJS or HTMLUnitDriver.

Earlier if you want to achieve headless test execution in Chrome, you had to use Xvfb (short for X virtual framebuffer). This is an in-memory display server for Linux which enables you to run graphical applications without a display.

But Google has released updates to chrome to achieve an inbuilt Headless test execution without using Xvfb or any other third party tools using a real browser – Chrome. This is available on Mac and Linux Os from Chrome 59. If you are using Windows, you need up update your Chrome version to 60 or above to achieve headless execution.

How to do Headless Test Execution: Its simple. All you have to do is add a chrome options argument before initializing your driver.

ChromeOptions options = new ChromeOptions();
WebDriver driver = new ChromeDriver(options);


Selenium 3.0 Officially Out Now

Yes. This is one of the major and stable release in the recent times since 2.0 got released around 5 years back.

What’s new we can expect: For users who are using WebDriver API’s this is gonna be just a drop in effortless replacement. But those who are using Selenium Grid may have to go through some minimal configuration changes.

In this version, they have removed the original Selenium Core implementation and replaced it with one backed by WebDriver. For people who are still using Selenium RC should have to undergo some maintenance because of this. This is the right time to migrate the code to WebDriver API’s.

Right from Selenium 3.0, all the major browser vendors are responsible for shipping their own implementation of WebDrivers for their browsers.

Starting with Safari-10, Apple started providing native support for WebDriver API’s. More details here – WebDriver Support in Safari 10

Google already started providing support for WebDriver API’s for Google Chrome using their ChromeDriver.

Even Microsoft when they came up with a new browser “EDGE” for Windows-10, they also came out with support for the WebDriver API’s – Microsoft WebDriver

Mozilla is doing major changes in the internals of Firefox browser to make it more stable and secure. So if you are using Firefox for your testing, you’ll need to use the GeckoDriver, which is an executable similar to the ChromeDriver. But Gecko is still in Alpha and you may have to face lot of issues with your automation w.r.t. Firefox.

The W3C specification for browser automation, based on Open Source WebDriver is still In-Progress and this is gonna reach “recommendation” status.

What you are waiting for? Just go and download the latest from Seleniumhq Downloads

Selenium 2016 Conference

Sometimes back I got an opportunity to attend the Selenium 2016 Conference in Bangalore India and it was a very good exposure for me to learn how people are doing things difference than me.

I thought of writing a detailed blog post for each and every session. But due to my busy schedule, I was not able to focus on my writing for the last few months.

So I am going to summarize some of the key take away sessions in this blog posts. The Slides and Videos are already there in the Conf Engine Site – Selenium 2016 Conference Slides & Videos.

The first and most favorite session was The Interviewer, Interviewee and vice-versa by Robin. Even-though he related this session with Selenium interviews, I can say each and everyone in our industry should watch this video once. Its not technical. But its funny and I enjoyed a lot. I highly recommend this for the interviewers to “What not to ask” in an interview.

The next one that I liked most is Automating the Responsive Web Design TestingIf you are developing a site/app and open to public, chances are there that our end users will access our app in a desktop, mobile or in a tablet. This session has some good amount of technical information related to Galen which helps us in automating testing for look and feel for our responsive websites.

Functional & Security Testing session is another interesting one which I already have tried out in one of my blog post Security Test Automation using Selenium and ZAP

Testing as a Container is a technical session which talks about leveraging the container technologies to solve the challenges of growing testing infrastructure and continuous delivery. We can learn how Docker works and why its important to our testing.

Automated Visual Testing – This session talks about modern comparison engines and visual testing tools like Applitools. It also talks about how we can use these image comparison tools to automation our baseline tests.

Test Data – Food for your Framework – This one talks about test data management. There is a myth in our automation that if there is a test data, it needs to be externalized in a config/excel or some other file other than source code. But this talk will break those beliefs.

Practical Tips & Tricks for Selenium by Dave the author of http://elementalselenium.com/ and https://seleniumguidebook.com/ gives us lot of advanced tips and tricks that will be helpful for our day to day automation activities.

Advanced Appium by Dan talks about Appium, the leading Mobile Test Automation Tool. This one is more technical and if you are learning any mobile test automation tool, at some point you have to go through this video.

Still there are plenty of interesting talks and Lightning Talks. You can go through all of these here – Selenium 2016 Conference Slides & Videos.

This is the Selenium 2016 Youtube Video Playlist where you can see all the videos.

Overall – Its lots & lots of key takeaways and learning from this conference. Eagerly looking forward to implement some of these ideas in my automation projects.

Virtualization using Vagrant for Selenium Tests

Say goodbye to “works on my machine” bugs.

Nobody likes to release software on a Friday. What if something breaks over the weekend? There’s nothing like debugging an issue in production when you’re doing some day drinking at a BBQ.

But you have nothing to worry about, right? I mean, you tested the application and it worked on *your* machine. Too bad that’s never good enough. Have fun working this weekend.

If only you had automated tests for your application (across all of the browsers and all the platforms you cared about). Then you could rest easy this weekend and enjoy the festivities.

Creating a robust and scalable execution environment for automation tests is a very essential phase. In automated tests for web, we actually need to cover different browsers and platforms.

In this documentation, we are going to see how to run our selenium tests in a Linux box using vagrant.

Basically I have a selenium test that works fine when running in different browsers in my Windows host machine. But I want to make sure that it is running fine on the browsers in a Linux box too. Instead of creating an actual VM and running my tests, I am simply going to create a light weight portable Linux box and I am going to run my selenium tests from my windows host machine.

Also every time that you want to run your tests in your local machine, it opens the browser on top of the other windows, preventing you from doing something else. Unless you use Phantomjs or HTMLUnitDriver, it is not possible to run Chrome or Firefox hidden, without disturbing you on your work. Now using Vagrant and VirtualBox, you only need to start the VM, and all your tests will be run into the VM. You can continue developing in the meanwhile!


Vagrant provides us easy to configure, reproducible, and portable work environments. Vagrant stands on top of VirtualBox, VMware and some other service providers. If you’re a developer, Vagrant will isolate dependencies and their configuration within a single disposable, consistent environment. If you’re a tester, Vagrant will help you to create light weight Virtual environments to run your tests against all the possible OS + Browser combinations.


  1. Download and install Oracle VM Virtualbox – https://www.virtualbox.org/wiki/Downloads
  2. Download and install Vagrant – http://www.vagrantup.com/downloads

From command line, type vagrant –v and make sure that you get the version number that you installed. Vagrant 1.7.21

Vagrant Boxes:

Boxes are the package format for Vagrant environments. A box can be used by anyone on any platform that Vagrant supports to bring up an identical working environment. The easiest way to use a box is to add a box from the publicly available catalog of Vagrant boxes for VirtualBox. We can also add and share our own customized boxes on this website.

I am going to use this box chef/ubuntu-14.04 – a standard Ubuntu 14.04 x64 base install.

Getting Started:

  1. Create a workspace directory to store the vagrant configuration file and shell scripts.

I have created “D:\Automation\Vagrant\Demo”

  1. From Command line go to your workspace and type vagrant init chef/ubuntu-14.04


  1. Make sure that the vagrant file is created in “D:\Automation\Vagrant\Demo”
  2. Run the command vagrant up

3If you are running this for first time, it may take some considerable amount of time to download the virtual box image and will put it under C:\Users\<username> \VirtualBox VMs\Demo_default_*

  1. Now the Ubuntu base image VM is up and running. Run the Command vagrant halt. This will power off the VM. Because we need to provision this VM with selenium related libraries, browsers and some utilities to run our selenium tests.
  2. Update the vagrant file in “D:\Automation\Vagrant\Demo” with the following content. Basically the default generated vagrant file has some lot of optional behaviors which are commented by default and you may enable it you need. But we require only the following content in a vagrant file.


Vagrant.configure(2) do |config|
   config.vm.box = "chef/ubuntu-14.04"
   config.vm.provision :shell, :path =&gt; "Provisioner.sh"
   config.vm.network :forwarded_port, guest:4444, host:4444

Basically what we are trying to do above is enabling port forwarding in the VM. Selenium server will use the 4444 port by default and we are forwarding that port from VM to host machine.

Also we are going to provision this VM using the “Provisioner.sh” shell script. Provisioners in Vagrant allow you to automatically install software, alter configurations, and more on the machine, as part of the vagrant up process. This is useful since boxes typically aren’t built perfectly for your use case. Of course, if you want to just use vagrant ssh and install the software by hand, that works. But by using the provisioning systems built-in to Vagrant, it automates the process so that it is repeatable. Most importantly, it requires no human interaction, so you can vagrant destroy and vagrant up and have a fully ready-to-go work environment with a single command.

Vagrant gives you multiple options for provisioning the machine, from simple shell scripts to more complex, industry-standard configuration management systems. If you’ve never used a configuration management system before, it is recommended you start with basic shell scripts for provisioning.

So to run a selenium test in a new plain OS, we need following things.

  • JDK/JRE – To run the selenium server.
  • Google chrome browser
  • ChromeDriver
  • Some utilities like ‘Unzip’ to extract the chromedriver zip file.
  • Selenium Standalone Server
  • Xvfbor X virtual framebuffer – To run tests in headless mode.

Xvfb is a display server implementing the X11 display server protocol. In contrast to other display servers, Xvfb performs all graphical operations in memory without showing any screen output. So actually when we run our tests in a VM box, we won’t be able to see any browser popping up. It means no GUI interactions are possible. So I am going to install all the above packages along with the dependencies using Provisioner.sh

  1. Create Provisioner.sh in the “D:\Automation\Vagrant\Demo” with the following content.

#!/usr/bin/env bash

# Set start time so we know how long the bootstrap takes

T="$(date +%s)"

#echo 'Updating'

sudo apt-get -y update

echo 'Installing Zip/Unzip'

sudo apt-get -y install zip unzip

echo 'Installing Google Chrome'

sudo apt-get -y install google-chrome-stable

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

sudo dpkg -i google-chrome-stable_current_amd64.deb

sudo apt-get -y install -f

echo 'Installing Google XVFB'

sudo apt-get -y install xvfb

sudo apt-get -y install -f

echo 'Installing JRE'

sudo apt-get -y install default-jdk

sudo apt-get -y install -f

echo 'Downloading and Moving the ChromeDriver/Selenium Server to /usr/local/bin'

cd /tmp

wget "http://chromedriver.storage.googleapis.com/2.8/chromedriver_linux64.zip"

wget "https://selenium.googlecode.com/files/selenium-server-standalone-2.35.0.jar"

unzip chromedriver_linux64.zip

mv chromedriver /usr/local/bin

mv selenium-server-standalone-2.35.0.jar /usr/local/bin

export DISPLAY=:10

cd /vagrant

echo "Starting Xvfb ..."

Xvfb :10 -screen 0 1366x768x24 -ac &

echo "Starting Google Chrome ..."

google-chrome --remote-debugging-port=9222 &

echo "Starting Selenium ..."

cd /usr/local/bin

java -jar selenium-server-standalone-2.35.0.jar

# Print how long the bootstrap script took to run

T="$(($(date +%s)-T))"

echo "Time bootstrap took: ${T} seconds"

  1. Run vagrant up –provision this will start the VM and then install the list of packages that we have added in the sh and also it will start the selenium server in 4444 port.


You can see this by navigating to http://localhost:4444/wd/hub/static/resource/hub.html from your windows host machine [As we have port forwarded].

Now run the following selenium test from any of your favorite IDE.

package test;

import java.net.URL;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;

public class HeadlessSession {	  
	  public static void main(String args[]) throws InterruptedException, Exception {	  
	  public static void HeadlessSessionId() throws Exception {
               DesiredCapabilities capabilities = new DesiredCapabilities();
	       WebDriver driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), capabilities);	  
		  try {				
			    String baseUrl = "http://www.google.com";
			    driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
			    System.out.println("Title : " + driver.getTitle());
			    String browserName = capabilities.getBrowserName().toLowerCase();
			    System.out.println("Browser : " + browserName);
		  } catch (Exception e) {
		  finally {

That’s it. Now you have successfully ran your tests in Chrome browser in a Linux box from your windows host machine.

Useful Vagrant Commands:

Command Action
vagrant up Power on the VM
vagrant halt Power off the VM
vagrant reload Restart
vagrant suspend Saving the VM state and sleep
vagrant resume Resuming the suspended VM
vagrant provision Provision the VM by running the Provisioner.sh or any other shell script mentioned in the vagrant file.
vagrant up –provision Power on with provision
vagrant reload –provision Restart with provision


  1. Easy to setup and maintain.
  2. It’s free.
  3. Able to clone the production/staging environment for our test execution.
  4. Light weight and portable.
  5. Support to provisioning scripts like Shell, Chef, and Puppet.
  6. Simple command line based workflow.
  7. Goodbye to ‘Works on my machine’ bugs.
  8. For selenium, you can able to run headless tests on your favorite browser.
  9. Create and destroy the VMs as needed.
  10. It works on all major platforms.
  11. We can also use Selenium Grid along with vagrant to run tests parallely across different VM’s.


  1. Base boxes for your choice of operating system might not be readily available.
  2. Little bit of time consuming to make vagrant VM up and running.
  3. Taking lot of time when provisioning the VM box. To avoid this, create your own base box by installing all the required dependencies by manually SSH into it and then use provisioning script only for starting the Selenium Server.
  4. Light weight. But looks fatty when considering the alternatives like Docker.
  5. It requires you to have a hard drive file that can be huge and it takes a lot of RAM.

Using Context in Soap-UI/Ready API External Script Library

In the recent post we have seen how to use the external script library to perform a session id extraction from an application.

One interesting thing I recently come across is, we can also use soap-UI context in our external script library.

Context is a built-in variable automatically made available by soapUI.

You can able to define/add your own properties of context objects
For those who are familiar with writing groovy test steps in Soap-UI definitely should heard of this term before.

//An example to set property and get property using soap-UI context.
context.setProperty("URL", "http://www.google.co.in/")

Lets see how this is going to be useful in our tests.

Let’s consider the same script that I have used before in my previous post
In that, have a look at the code written in “The Script Library to get the SessionID

The method session contains three parameters. URL, Username and Password.

If you are going to call this session method from inside your Soap-UI, you will need to pass the value for these three parameters.

Obviously, Things like URL, Username and Password will be usually stored in a Project Level Property to make it easy for configuration.

So you have to first get these three Project Level Property and then you have to call this external script library.

So if you have a close look at the section “How do we Call this Script inside our SoapUI test cases?” in this Post,

You can able to see that we have extracted the project level property in a variable and then we have used these variables as a parameter in when calling the session method.

def groovyUtils = new com.eviware.soapui.support.GroovyUtils( context );
String url = context.expand( '${#Project#AuthenticateRequestURL}' )
String username = "yourusername";
String password = "yourpassword";
Obj = new Headlessdriver();
String session = Obj.session(url, username, password);

Do we really need to do this every time when we call the external script library? Answer is NO.

We can still use the ‘context’ variable in our external script library also. So I have updated the Headlessdriver.groovy as follows.

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver; 
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.ui.ExpectedCondition;
import org.openqa.selenium.support.ui.WebDriverWait;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.phantomjs.PhantomJSDriver;
import org.openqa.selenium.phantomjs.PhantomJSDriverService;

class Headlessdriver {

	String url;
	String username;
	String password;
	String phantomPath;

	def Headlessdriver( context ) {        
        this.url = context.expand( '${#Project#URL}' );
		this.username = context.expand( '${#Project#username}' );
		this.password = context.expand( '${#Project#password}' );
		this.phantomPath = context.expand( '${#Project#phantomPath}' );

	String session () {

		DesiredCapabilities caps = new DesiredCapabilities();
		// caps.setCapability("phantomjs.binary.path", "ext\\phantomjs.exe");
		caps.setCapability("phantomjs.binary.path", phantomPath);
		String[] args = [ "--ignore-ssl-errors=yes" ];
		caps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, args);
		WebDriver driver = new PhantomJSDriver(caps);
		driver.manage().timeouts().implicitlyWait(60L, TimeUnit.SECONDS);
		String sessionIdFull = driver.manage().getCookieNamed("sessionId").toString();
		String SessionId = sessionIdFull.split("; ")[0];
		return SessionId;

Now you don’t need to extract the project level property in your groovy script test step before calling this external script library.
Calling this Headlessdriver.session() method is simple as below.

Obj = new Headlessdriver( context );
String session = Obj.session();

SoapUI Married with Selenium

This includes the steps to add up Selenium related libraries to be added to the SOAPUI installation

Step-by-step guide to Setup Selenium Library

  1. Download selenium-server-standalone.jar which includes the Selenium Related libraries.
  2. Copy And paste the downloaded jar file to the SOAPUI installation path, ext folder .
    Path : \ bin \ ext
  3. Download the phantomjs.exe Version-2.0 and Paste the exe file in  \bin\ext directory
  4. Restart Ready-API.

With the above steps, you have added the Selenium library against your SOAPUI installation.

Why do we need a Selenium Library.

  1. There may be instances where we may require to interact with UI to get some data for API automation.
    e.g getting Session-id for automating secure API automation.
  2. Cases to perform end to end testing.

The Script Library to get the SessionID:

I have developed the following Class which will get the the SessionId in headless mode or ghost/daemon mode.

We call it as headless mode or ghost/daemon mode because this program actually runs in the background without a user interface or simply the browser won’t show up when running this test.

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.ui.ExpectedCondition;
import org.openqa.selenium.support.ui.WebDriverWait;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.phantomjs.PhantomJSDriver;
import org.openqa.selenium.phantomjs.PhantomJSDriverService;
class Headlessdriver {
    String session (String url, String username, String password) {
    DesiredCapabilities caps = new DesiredCapabilities();
    caps.setCapability("phantomjs.binary.path", "ext\\phantomjs.exe");
    String[] args = [ "--ignore-ssl-errors=yes" ];
    caps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, args);
    //System.setProperty("webdriver.chrome.driver", ".//chromedriver.exe");
    WebDriver driver = new PhantomJSDriver(caps);
    driver.manage().timeouts().implicitlyWait(60L, TimeUnit.SECONDS);
    // Sign in on the CAS login page
    String sessionIdFull = driver.manage().getCookieNamed("sessionId").toString();
    String SessionId = sessionIdFull.split("; ")[0];
    return SessionId;

Save this script as “Headlessdriver.groovy”.

One interesting option in SoapUI is you can define your own script libraries for external groovy scripts.

By default the Script Library points to “\bin\scripts”

You can see that by going to File->Preferences->Ready! API->Script Library

But if you are using source control, then it will be convenient for you to check in the external groovy scripts inside your project.

So what I have done is, I have created a directory called “Scripts” inside my project. Now you can change the above preferences to point the scripts located in your script library.

But in the recent versions, you have to configure this in your Project Level also using relative path.

Click on you “Project” Go to “Project Properties” tab and Change “Resource Root” to “${projectDir}”

Change “Script Library” to “${projectDir}\API-Tests\Scripts”

How do we Call this Script inside our SoapUI test cases?

You can Call this Script from your “Test Case SetUp” or in your “Test Suite SetUp” or from “Groovy Script Test Step”.

def groovyUtils = new com.eviware.soapui.support.GroovyUtils( context );
String url = context.expand( '${#Project#AuthenticateRequestURL}' )
String username = "yourusername";
String password = "yourpassword";
Obj = new Headlessdriver();
String session = Obj.session(url, username, password);
//When Calling from SetUp
testSuite.setPropertyValue("SessionId", session);
//When Calling from Groovy Script Test Step
testRunner.testCase.testSuite.setPropertyValue("SessionId", session); 

The above script will create/update a Property called “SessionId” in the your Test Suite.

You can use this Session information in your header for hitting Secured API’s.

Create a Header Named “Cookie” with Value ${#TestSuite#SessionId} in your SoapUI `http requests.

SoapUI Get SessionID

The whole purpose is, I have to automate testing for some Secured API’s, which need Session Id of my application to send request and to receive expected response.

I am using Soap-UI for my API Testing.

Initially I thought of using SoapUI to achieve the same. But my application uses SAML requests and do lot of redirection before getting into the Login Page.

So my alternative workaround is to go for the some other frameworks to fetch this session Id.

I have already discussed the same in the following post.


But in the above mentioned Post, I was using the FirefoxDriver. This will trigger the Firefox browser all the time and then it will fetch me the session information.

I somehow felt inside that this is not a proper workaround and always searched for the better alternative solutions.

Now I just want to let you know something that I have tried earlier to get the Session Cookie in headless mode without a browser.

Earlier I have tried out a solution to use the HTMLUnitDriver instead of FirefoxDriver to get the session information with a headless browser.

Initially I got lot of exceptions with earlier version of selenium standalone server.

But this works perfectly with the latest version of selenium. I really don’t know why.

Now the updated script will run in daemon mode and will fetch you the session id without a browser.

Obviously if you use HTMLUnitDriver, it will always throw severe bunch of warnings that you may not need.

All we need is just the session information of the application.

So, somehow I found out a way to turn off the HTMLUnitDriver logging just to avoid these bunch of severe warnings and unwanted information.

The following method will get you the SessionId of a web application.

I just pasted the method here. Of course you guyz may need to do some modification based on your needs.

You need to add the latest version of Selenium-Standalone-Server in your class path for this method to work.

public static void HeadlessSessionId() throws Exception {
		  WebDriver driver = new HtmlUnitDriver(true);
		  try {		  	
			  	LogFactory.getFactory().setAttribute("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");
				String domainString = ServiceEndPoint;		
			    String baseUrl = domainString;
			    driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
			    driver.get(baseUrl + "/Demo/");			    
			    String SessionId = driver.manage().getCookieNamed("sessionId").toString();			    
			    domainString = domainString.replaceAll("http://", "domain=");
			    String SessionID = ((SessionId).replaceAll("path=/;", " ")).replaceAll(domainString, "").replaceAll("Cookie: ", ""); 
			    Cookie = SessionID;
		  finally {

So That’s it. We have the session information of our application in Hand. Now how we are going to integrate this with our SoapUI?

During the development phase/when running my tests manually from SoapUI GUI, I have a project level property defined in SoapUI called “Cookie”.

And I will manually replace that project property with the session information that I got from my selenium script.

Later I will use this Project level Property in my SoapUI test cases/ test steps.

But as a automation engineer, at some point, I may want to add these tests in my Continuous Integration Server to run these tests on a nightly Builds.

At that time, there should not be any manual intervention in my automated tests.

So lets see how I actually integrated the session that I am getting it from Selenium with SoapUI.

All we need to do is simple.

Instead of printing the session Id, write it in a property file. I used to write this session Id in “SoapUIProjectProerties.props”

Export the entire Java project into a runnable Jar file.

If you are using any IDE like Eclipse, it is not that difficult task. Then create a Batch file that will do the following steps.

1. First run the Jar File and this will write the SessionID into that Property File.
2. Use SoapUI TestRunner Batch from Command Line and Specify it to load the Properties from the “SoapUIProjectProerties.props”

SET SOAPUI_TEST_RUNNER="C:\Program Files (x86)\SmartBear\SoapUI-5.0.0\bin\"
java -jar HeadlessSessionId.jar
%SOAPUI_TEST_RUNNER%testrunner.bat -sTestSuiteName -r -a -j -f"Reports" -i Project-Name.xml 

Now during the runtime, the session Id property defined in the property file will be used by SoapUI for hitting secured API’s.

I hope this solution might be helpful for your when automating secured API’s which need session Id.

My Colleague is working on another interesting solution to rewrite the whole Java program as a ‘Groovy Test Script’ in SoapUI.

I will let you know if that works. Thanks.