Check for broken links on your website using Postman

If you are using Postman for your API Testing, then you can also you the same to automatically crawl all the pages on our website and check every link for a healthy HTTP status code.

This can be achieved using 2 simple API requests in Postman.

First lets create a new Collection and an Environment in Postman, where you can specify

  • root_url
  • start_url

Specify the values for the root_url and start_url.

root_url as

start_url as

Create a simple request with Get method and enter url as {{start_url}} and in the Tests tab enter the following code.

// set environment variables to default values
postman.setEnvironmentVariable('links', '[]');
postman.setEnvironmentVariable('url', postman.getEnvironmentVariable('start_url'));
postman.setEnvironmentVariable('index', -1);



Create a second request – Get method and enter URL as {{url}} and in the Tests tab enter the following code.

// Tests and custom scripts can be written in JavaScript.

// get environment variables
var start_url = postman.getEnvironmentVariable('start_url');
var root_url = postman.getEnvironmentVariable('root_url');
var links = JSON.parse(postman.getEnvironmentVariable('links'));
var url = postman.getEnvironmentVariable('url');
var index = parseInt(postman.getEnvironmentVariable('index'));

// increment index counter to access links in array to check
index = index + 1;

// test if link works
if (responseCode.code > 400) {
 console.log("This link is broken: ", url);
 tests["Link works"] = false;
} else {
 tests["Link works"] = true;

// if the current url includes the start_url, then this is an internal link and we should crawl it for more links
if (url.includes(start_url)) {

 // load the response body as HTML using cheerio, get the <a> tags
 var $ = cheerio.load(responseBody);

 $('a').each(function (index) {

 var link = $(this).attr('href');

 // add links to the links array if not already in there
 // if you have additional links you would like to exclude, for example, ads, you can add this criteria as well
 if (!links.includes(link)) {

// if we've gone through all the links, return early
if (links.length - 1 === index) {
 console.log('no more links to check');

// if link is a relative one, prepend with root_url
url = links[index]
if (! /^https?:\/\//.test(url)) {
 url = root_url + url;

// update environment variable values
postman.setEnvironmentVariable("links", JSON.stringify(links));
postman.setEnvironmentVariable("url", url);
postman.setEnvironmentVariable("index", index);

// continue calling the same request until all links are checked
postman.setNextRequest('Check URL');

Now Open “Runner” Select the Collection, Select the Environment and Click on Start Run Button.


You can see Postman in action crawling all the links until there are no more links to check.

You can also simply download the Postman Collection and import it into your Postman.


SoapUI/Ready API Test Suite and Test Case Count

If we have a very large number of tests in our project, and if you want to calculate the number of test suites/ test cases out of it, simply use the below script as a Groovy Script Test Step in your SoapUI.

def totalTestSuiteCount = 0;
def enabledTestSuiteCount = 0;
def disabledTestSuiteCount = 0;

totalTestSuiteCount = testRunner.testCase.testSuite.project.getTestSuiteCount();

for(i = 0; i < totalTestSuiteCount; i++) {
flag = testRunner.testCase.testSuite.project.getTestSuiteAt(i).isDisabled();
enabledTestSuiteCount = enabledTestSuiteCount + 1;
disabledTestSuiteCount = disabledTestSuiteCount + 1;
} "Total Test Suite Count : " + totalTestSuiteCount; "Total Enabled/Active Test Suite Count : " + enabledTestSuiteCount; "Total Disabled/Reusable Test Suite Count : " + disabledTestSuiteCount;
def totalTestCaseCount = 0;
def enabledTestCaseCount = 0;
def disabledTestCaseCount = 0;

for(i = 0; i < totalTestSuiteCount; i++) {
flag = testRunner.testCase.testSuite.project.getTestSuiteAt(i).isDisabled();
if(!flag) {
totalTestCaseCount = totalTestCaseCount + testRunner.testCase.testSuite.project.getTestSuiteAt(i).getTestCaseCount();

for(i = 0; i < totalTestSuiteCount; i++) {
flag = testRunner.testCase.testSuite.project.getTestSuiteAt(i).isDisabled();
if(!flag) {
tccount = testRunner.testCase.testSuite.project.getTestSuiteAt(i).getTestCaseCount();
for(j = 0; j < tccount; j++) {
tcflag = testRunner.testCase.testSuite.project.getTestSuiteAt(i).getTestCaseAt(j).isDisabled();
enabledTestCaseCount = enabledTestCaseCount + 1;
disabledTestCaseCount = disabledTestCaseCount + 1;
} "Total Test Case Count : " + totalTestCaseCount; "Total Enabled/Active Test Case Count : " + enabledTestCaseCount; "Total Disabled/Reusable Test Case Count : " + disabledTestCaseCount;

I am here under an impression that, we should exclude the disabled test suites/cases from the count.

Because, in general we used to disable the test suite/case which is not in a proper working condition or if it is acting as a reusable library.

In my case, I have the habit of creating the reusable components/libraries as a test suite/test case and finally I will disable them to exclude those things from normal execution flow.

So I thought, those things should be omitted from the count when taking the statistics.

The next thing I am looking out for is to cover the datasource/datasource loop count also in number of test cases.

Because we are combining so many similar functional test scenarios into one single test case using datasource/datasource loop test steps.

But finally when getting the count, this will also be calculated as one single test case. I don’t want this to be a single test case. If my data source loop is getting executed 10 times, then my test case count also should be 10 instead of 1.

Using Context in Soap-UI/Ready API External Script Library

In the recent post we have seen how to use the external script library to perform a session id extraction from an application.

One interesting thing I recently come across is, we can also use soap-UI context in our external script library.

Context is a built-in variable automatically made available by soapUI.

You can able to define/add your own properties of context objects
For those who are familiar with writing groovy test steps in Soap-UI definitely should heard of this term before.

//An example to set property and get property using soap-UI context.
context.setProperty("URL", "")"URL"))

Lets see how this is going to be useful in our tests.

Let’s consider the same script that I have used before in my previous post
In that, have a look at the code written in “The Script Library to get the SessionID

The method session contains three parameters. URL, Username and Password.

If you are going to call this session method from inside your Soap-UI, you will need to pass the value for these three parameters.

Obviously, Things like URL, Username and Password will be usually stored in a Project Level Property to make it easy for configuration.

So you have to first get these three Project Level Property and then you have to call this external script library.

So if you have a close look at the section “How do we Call this Script inside our SoapUI test cases?” in this Post,

You can able to see that we have extracted the project level property in a variable and then we have used these variables as a parameter in when calling the session method.

def groovyUtils = new context );
String url = context.expand( '${#Project#AuthenticateRequestURL}' )
String username = "yourusername";
String password = "yourpassword";
Obj = new Headlessdriver();
String session = Obj.session(url, username, password);

Do we really need to do this every time when we call the external script library? Answer is NO.

We can still use the ‘context’ variable in our external script library also. So I have updated the Headlessdriver.groovy as follows.

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver; 
import org.openqa.selenium.WebElement;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.phantomjs.PhantomJSDriver;
import org.openqa.selenium.phantomjs.PhantomJSDriverService;

class Headlessdriver {

	String url;
	String username;
	String password;
	String phantomPath;

	def Headlessdriver( context ) {        
        this.url = context.expand( '${#Project#URL}' );
		this.username = context.expand( '${#Project#username}' );
		this.password = context.expand( '${#Project#password}' );
		this.phantomPath = context.expand( '${#Project#phantomPath}' );

	String session () {

		DesiredCapabilities caps = new DesiredCapabilities();
		// caps.setCapability("phantomjs.binary.path", "ext\\phantomjs.exe");
		caps.setCapability("phantomjs.binary.path", phantomPath);
		String[] args = [ "--ignore-ssl-errors=yes" ];
		caps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, args);
		WebDriver driver = new PhantomJSDriver(caps);
		driver.manage().timeouts().implicitlyWait(60L, TimeUnit.SECONDS);
		String sessionIdFull = driver.manage().getCookieNamed("sessionId").toString();
		String SessionId = sessionIdFull.split("; ")[0];
		return SessionId;

Now you don’t need to extract the project level property in your groovy script test step before calling this external script library.
Calling this Headlessdriver.session() method is simple as below.

Obj = new Headlessdriver( context );
String session = Obj.session();

SoapUI Married with Selenium

This includes the steps to add up Selenium related libraries to be added to the SOAPUI installation

Step-by-step guide to Setup Selenium Library

  1. Download selenium-server-standalone.jar which includes the Selenium Related libraries.
  2. Copy And paste the downloaded jar file to the SOAPUI installation path, ext folder .
    Path : \ bin \ ext
  3. Download the phantomjs.exe Version-2.0 and Paste the exe file in  \bin\ext directory
  4. Restart Ready-API.

With the above steps, you have added the Selenium library against your SOAPUI installation.

Why do we need a Selenium Library.

  1. There may be instances where we may require to interact with UI to get some data for API automation.
    e.g getting Session-id for automating secure API automation.
  2. Cases to perform end to end testing.

The Script Library to get the SessionID:

I have developed the following Class which will get the the SessionId in headless mode or ghost/daemon mode.

We call it as headless mode or ghost/daemon mode because this program actually runs in the background without a user interface or simply the browser won’t show up when running this test.

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.phantomjs.PhantomJSDriver;
import org.openqa.selenium.phantomjs.PhantomJSDriverService;
class Headlessdriver {
    String session (String url, String username, String password) {
    DesiredCapabilities caps = new DesiredCapabilities();
    caps.setCapability("phantomjs.binary.path", "ext\\phantomjs.exe");
    String[] args = [ "--ignore-ssl-errors=yes" ];
    caps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, args);
    //System.setProperty("", ".//chromedriver.exe");
    WebDriver driver = new PhantomJSDriver(caps);
    driver.manage().timeouts().implicitlyWait(60L, TimeUnit.SECONDS);
    // Sign in on the CAS login page
    String sessionIdFull = driver.manage().getCookieNamed("sessionId").toString();
    String SessionId = sessionIdFull.split("; ")[0];
    return SessionId;

Save this script as “Headlessdriver.groovy”.

One interesting option in SoapUI is you can define your own script libraries for external groovy scripts.

By default the Script Library points to “\bin\scripts”

You can see that by going to File->Preferences->Ready! API->Script Library

But if you are using source control, then it will be convenient for you to check in the external groovy scripts inside your project.

So what I have done is, I have created a directory called “Scripts” inside my project. Now you can change the above preferences to point the scripts located in your script library.

But in the recent versions, you have to configure this in your Project Level also using relative path.

Click on you “Project” Go to “Project Properties” tab and Change “Resource Root” to “${projectDir}”

Change “Script Library” to “${projectDir}\API-Tests\Scripts”

How do we Call this Script inside our SoapUI test cases?

You can Call this Script from your “Test Case SetUp” or in your “Test Suite SetUp” or from “Groovy Script Test Step”.

def groovyUtils = new context );
String url = context.expand( '${#Project#AuthenticateRequestURL}' )
String username = "yourusername";
String password = "yourpassword";
Obj = new Headlessdriver();
String session = Obj.session(url, username, password);
//When Calling from SetUp
testSuite.setPropertyValue("SessionId", session);
//When Calling from Groovy Script Test Step
testRunner.testCase.testSuite.setPropertyValue("SessionId", session); 

The above script will create/update a Property called “SessionId” in the your Test Suite.

You can use this Session information in your header for hitting Secured API’s.

Create a Header Named “Cookie” with Value ${#TestSuite#SessionId} in your SoapUI `http requests.

SoapUI Get SessionID

The whole purpose is, I have to automate testing for some Secured API’s, which need Session Id of my application to send request and to receive expected response.

I am using Soap-UI for my API Testing.

Initially I thought of using SoapUI to achieve the same. But my application uses SAML requests and do lot of redirection before getting into the Login Page.

So my alternative workaround is to go for the some other frameworks to fetch this session Id.

I have already discussed the same in the following post.

But in the above mentioned Post, I was using the FirefoxDriver. This will trigger the Firefox browser all the time and then it will fetch me the session information.

I somehow felt inside that this is not a proper workaround and always searched for the better alternative solutions.

Now I just want to let you know something that I have tried earlier to get the Session Cookie in headless mode without a browser.

Earlier I have tried out a solution to use the HTMLUnitDriver instead of FirefoxDriver to get the session information with a headless browser.

Initially I got lot of exceptions with earlier version of selenium standalone server.

But this works perfectly with the latest version of selenium. I really don’t know why.

Now the updated script will run in daemon mode and will fetch you the session id without a browser.

Obviously if you use HTMLUnitDriver, it will always throw severe bunch of warnings that you may not need.

All we need is just the session information of the application.

So, somehow I found out a way to turn off the HTMLUnitDriver logging just to avoid these bunch of severe warnings and unwanted information.

The following method will get you the SessionId of a web application.

I just pasted the method here. Of course you guyz may need to do some modification based on your needs.

You need to add the latest version of Selenium-Standalone-Server in your class path for this method to work.

public static void HeadlessSessionId() throws Exception {
		  WebDriver driver = new HtmlUnitDriver(true);
		  try {		  	
			  	LogFactory.getFactory().setAttribute("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");
				String domainString = ServiceEndPoint;		
			    String baseUrl = domainString;
			    driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
			    driver.get(baseUrl + "/Demo/");			    
			    String SessionId = driver.manage().getCookieNamed("sessionId").toString();			    
			    domainString = domainString.replaceAll("http://", "domain=");
			    String SessionID = ((SessionId).replaceAll("path=/;", " ")).replaceAll(domainString, "").replaceAll("Cookie: ", ""); 
			    Cookie = SessionID;
		  finally {

So That’s it. We have the session information of our application in Hand. Now how we are going to integrate this with our SoapUI?

During the development phase/when running my tests manually from SoapUI GUI, I have a project level property defined in SoapUI called “Cookie”.

And I will manually replace that project property with the session information that I got from my selenium script.

Later I will use this Project level Property in my SoapUI test cases/ test steps.

But as a automation engineer, at some point, I may want to add these tests in my Continuous Integration Server to run these tests on a nightly Builds.

At that time, there should not be any manual intervention in my automated tests.

So lets see how I actually integrated the session that I am getting it from Selenium with SoapUI.

All we need to do is simple.

Instead of printing the session Id, write it in a property file. I used to write this session Id in “SoapUIProjectProerties.props”

Export the entire Java project into a runnable Jar file.

If you are using any IDE like Eclipse, it is not that difficult task. Then create a Batch file that will do the following steps.

1. First run the Jar File and this will write the SessionID into that Property File.
2. Use SoapUI TestRunner Batch from Command Line and Specify it to load the Properties from the “SoapUIProjectProerties.props”

SET SOAPUI_TEST_RUNNER="C:\Program Files (x86)\SmartBear\SoapUI-5.0.0\bin\"
java -jar HeadlessSessionId.jar
%SOAPUI_TEST_RUNNER%testrunner.bat -sTestSuiteName -r -a -j -f"Reports" -i Project-Name.xml

Now during the runtime, the session Id property defined in the property file will be used by SoapUI for hitting secured API’s.

I hope this solution might be helpful for your when automating secured API’s which need session Id.

My Colleague is working on another interesting solution to rewrite the whole Java program as a ‘Groovy Test Script’ in SoapUI.

I will let you know if that works. Thanks.

SoapUI – Properties using Groovy

def sdf = new java.text.SimpleDateFormat("yyyy-MM-dd hh:mm:ss") (sdf.format( new Date()))
def dff = new java.text.SimpleDateFormat("yyyy-dd-MM hh:mm:ss") (dff.format( new Date()))
def dtt = new java.text.SimpleDateFormat("yyyy-dd-MM") (dtt.format( new Date()))
//TO get the Project Name
def username = (username)
//To get the Test Suite Name
def username1 = (username1)
//TO get the Test Case Name
def username2 = (username2)
//TO get the Test Step Name
def username3 = context.getCurrentStep().getLabel() (username3)
testRunner.testCase.setPropertyValue( "MyProp", 'someValue' )
def testCaseProperty = testRunner.testCase.getPropertyValue( "MyProp" )
testRunner.testCase.testSuite.setPropertyValue( "MyProp", 'someValue' )
def testSuiteProperty = testRunner.testCase.testSuite.getPropertyValue( "MyProp" )
testRunner.testCase.testSuite.project.setPropertyValue( "MyProp", 'someValue' )
def projectProperty = testRunner.testCase.testSuite.project.getPropertyValue( "MyProp" )

Sending HTTP Requests using Apache HTTP Client

Sending HTTP Requests using Apache HTTP Client:

Below is the sample code to test end points using Apache Http Client in Java
This is a GET Method.

package org.apache.http.examples.client;

import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.HttpClientBuilder;

public class GetLibrary {
	public static void main(String args[]) throws Exception {
		String url = "";
		HttpClient client = HttpClientBuilder.create().build();
		HttpGet request = new HttpGet(url);		
		request.addHeader("User-Agent", "USER-AGENT");
		HttpResponse response = client.execute(request);	 
		System.out.println("Response Code : " 
	                + response.getStatusLine().getStatusCode());	 
		BufferedReader rd = new BufferedReader(
			new InputStreamReader(response.getEntity().getContent()));	 
		StringBuffer result = new StringBuffer();
		String line = "";
		while ((line = rd.readLine()) != null) {