Java

Selenium Web Scraping Hello World with Java

Selenium is, primarily, used for automating web applications for testing purposes. However, it could also be used for doing web scraping if required. In this post, you would learn about how to use Selenium for Web Scraping using Java. The following are some of the topics described in this article.

  • Maven project with Selenium Artifact Entry in POM.xml
  • Steps for Writing Web scraping Automation

Maven Project with Selenium Artifact Entry in POM.xml

Create a Maven Project within your Eclipse IDE and place the following entry in pom.xml file. Get the latest artifacts from Maven Page for Selenium Java page.

<!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java -->
<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>3.11.0</version>
</dependency>

The above would result in the download of appropriate Java libraries (JAR files) to run Selenium program. The following screenshot represents the downloads and where to find them.

Steps for Writing Web Scraping Automation

The following is the sample code for feeding the inputs to Indeed home page (https://www.indeed.co.in) for searching jobs in a particular location. The code given below automates the job search by providing inputs for “What” and “Where” field and clicking on the “Find Jobs” button.

import java.io.File;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;

/**
 * Indeed Job Search Hello World Program
 *
 */public class IndeedJobSearch 
{
    public static void main( String[] args ) throws InterruptedException
    {
        //
        // Set the path of the driver to driver executable. For Chrome, set the properties as following:
        //
        File file = new File("D:/softwares/selenium/chromedriver_win32/chromedriver.exe");
        System.setProperty("webdriver.chrome.driver", file.getAbsolutePath());
        // Create a Chrome Web Driver
        //
        WebDriver driver = new ChromeDriver();
        //
        // Open the Indeed.com homepage
        //
        driver.get("https://www.indeed.co.in");
        //
        // Enter the keyword "Java" in the field, What
        //
        driver.findElement(By.id("what")).clear();
        Thread.sleep(2000);
        driver.findElement(By.id("what")).sendKeys("Java");
        //
        // Enter the keyword, "Hyderabad" in the field, Where
        //
        driver.findElement(By.id("where")).clear();
        Thread.sleep(2000);
        driver.findElement(By.id("where")).sendKeys("Hyderabad");
        //
        // Click the FindJobs button for searching
        //
        driver.findElement(By.id("fj")).click();
        //
        // Print the information from the new page
        //
        System.out.println("Page Title:" + driver.getTitle());
        System.out.println("Jobs Count: " + driver.findElement(By.id("searchCount")).getText());
        Thread.sleep(2000);
        //
        // Close the browser
        //
        driver.close();
    }
}

Pay attention to some of the following in the code given above.

  • Configure Driver Executable Path: Set the path of the driver to driver executable. The drivers can be downloaded from Selenium Downloads page. The drivers for different browsers are listed under the heading, Third Party Drivers, Bindings, and Plugins. The following code represents the usage of Chrome Driver:
    File file = new File("D:/softwares/selenium/chromedriver_win32/chromedriver.exe");
    System.setProperty("webdriver.chrome.driver", file.getAbsolutePath());
    
  • Create Web Driver Instance: Create an appropriate web driver instance by making use of an appropriate implementation class. The implementation classes represent the web browsers. The following are some of the classes supported for different web browsers:
    • ChromeDriver
    • FirefoxDriver
    • InternetExplorerDriver
    • OperaDriver
    • SafariDriver
    • EdgeDriver
    • EventFiringWebDriver
    • RemoteWebDriver,
      The following code represents the creating of an instance of Web Driver:
      WebDriver driver = new ChromeDriver();
      
  • Open the Website: Open the website’s web page to be tested
    driver.get("https://www.indeed.co.in");
    
  • Perform Automation Steps: The following are the automation steps:
    • Provide the inputs. In the example given above, the input fields are cleared first, and then, data is entered into the field. Failing to do so would append the data to already existing text in the field, if any. Note that Thread.sleep() is used to pause the steps as appropriate.
      //
      // Enter the keyword "Java" in the field, What
      //
      driver.findElement(By.id("what")).clear();
      Thread.sleep(2000);
      driver.findElement(By.id("what")).sendKeys("Java");
      //
      // Enter the keyword, "Hyderabad" in the field, Where
      //
      driver.findElement(By.id("where")).clear();
      Thread.sleep(2000);
      driver.findElement(By.id("where")).sendKeys("Hyderabad");
      
    • Submit and get the results page. The following command would result in submitting the inputs to be submitted.
      driver.findElement(By.id("fj")).click();
      
    • Analyze the results; The following are sample commands:
      System.out.println("Page Title:" + driver.getTitle());
      System.out.println("Jobs Count: " + driver.findElement(By.id("searchCount")).getText());
      
  • Close the browser.
    driver.close();
    

Further Reading/References

Summary

In this post, you learned about how to use Selenium for Web Scraping using Java programming language.

Did you find this article useful? Do you have any questions or suggestions about this article? Leave a comment and ask your questions and I shall do my best to address your queries.

Ajitesh Kumar

I have been recently working in the area of Data analytics including Data Science and Machine Learning / Deep Learning. I am also passionate about different technologies including programming languages such as Java/JEE, Javascript, Python, R, Julia, etc, and technologies such as Blockchain, mobile computing, cloud-native technologies, application security, cloud computing platforms, big data, etc. I would love to connect with you on Linkedin. Check out my latest book titled as First Principles Thinking: Building winning products using first principles thinking.

Recent Posts

Agentic Reasoning Design Patterns in AI: Examples

In recent years, artificial intelligence (AI) has evolved to include more sophisticated and capable agents,…

1 month ago

LLMs for Adaptive Learning & Personalized Education

Adaptive learning helps in tailoring learning experiences to fit the unique needs of each student.…

2 months ago

Sparse Mixture of Experts (MoE) Models: Examples

With the increasing demand for more powerful machine learning (ML) systems that can handle diverse…

2 months ago

Anxiety Disorder Detection & Machine Learning Techniques

Anxiety is a common mental health condition that affects millions of people around the world.…

2 months ago

Confounder Features & Machine Learning Models: Examples

In machine learning, confounder features or variables can significantly affect the accuracy and validity of…

2 months ago

Credit Card Fraud Detection & Machine Learning

Last updated: 26 Sept, 2024 Credit card fraud detection is a major concern for credit…

2 months ago