Selenium is, primarily, used for automating web applications for testing purposes. However, it could also be used for doing web scraping if required. In this post, you would learn about how to use Selenium for Web Scraping using Java. The following are some of the topics described in this article.
- Maven project with Selenium Artifact Entry in POM.xml
- Steps for Writing Web scraping Automation
Maven Project with Selenium Artifact Entry in POM.xml
Create a Maven Project within your Eclipse IDE and place the following entry in pom.xml file. Get the latest artifacts from Maven Page for Selenium Java page.
<!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java -->
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.11.0</version>
</dependency>
The above would result in the download of appropriate Java libraries (JAR files) to run Selenium program. The following screenshot represents the downloads and where to find them.
Steps for Writing Web Scraping Automation
The following is the sample code for feeding the inputs to Indeed home page (https://www.indeed.co.in) for searching jobs in a particular location. The code given below automates the job search by providing inputs for “What” and “Where” field and clicking on the “Find Jobs” button.
import java.io.File;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
/**
* Indeed Job Search Hello World Program
*
*/
public class IndeedJobSearch
{
public static void main( String[] args ) throws InterruptedException
{
//
// Set the path of the driver to driver executable. For Chrome, set the properties as following:
//
File file = new File("D:/softwares/selenium/chromedriver_win32/chromedriver.exe");
System.setProperty("webdriver.chrome.driver", file.getAbsolutePath());
// Create a Chrome Web Driver
//
WebDriver driver = new ChromeDriver();
//
// Open the Indeed.com homepage
//
driver.get("https://www.indeed.co.in");
//
// Enter the keyword "Java" in the field, What
//
driver.findElement(By.id("what")).clear();
Thread.sleep(2000);
driver.findElement(By.id("what")).sendKeys("Java");
//
// Enter the keyword, "Hyderabad" in the field, Where
//
driver.findElement(By.id("where")).clear();
Thread.sleep(2000);
driver.findElement(By.id("where")).sendKeys("Hyderabad");
//
// Click the FindJobs button for searching
//
driver.findElement(By.id("fj")).click();
//
// Print the information from the new page
//
System.out.println("Page Title:" + driver.getTitle());
System.out.println("Jobs Count: " + driver.findElement(By.id("searchCount")).getText());
Thread.sleep(2000);
//
// Close the browser
//
driver.close();
}
}
Pay attention to some of the following in the code given above.
- Configure Driver Executable Path: Set the path of the driver to driver executable. The drivers can be downloaded from Selenium Downloads page. The drivers for different browsers are listed under the heading, Third Party Drivers, Bindings, and Plugins. The following code represents the usage of Chrome Driver:
File file = new File("D:/softwares/selenium/chromedriver_win32/chromedriver.exe"); System.setProperty("webdriver.chrome.driver", file.getAbsolutePath()); - Create Web Driver Instance: Create an appropriate web driver instance by making use of an appropriate implementation class. The implementation classes represent the web browsers. The following are some of the classes supported for different web browsers:
- ChromeDriver
- FirefoxDriver
- InternetExplorerDriver
- OperaDriver
- SafariDriver
- EdgeDriver
- EventFiringWebDriver
- RemoteWebDriver,
The following code represents the creating of an instance of Web Driver:WebDriver driver = new ChromeDriver();
- Open the Website: Open the website’s web page to be tested
driver.get("https://www.indeed.co.in"); - Perform Automation Steps: The following are the automation steps:
- Provide the inputs. In the example given above, the input fields are cleared first, and then, data is entered into the field. Failing to do so would append the data to already existing text in the field, if any. Note that Thread.sleep() is used to pause the steps as appropriate.
// // Enter the keyword "Java" in the field, What // driver.findElement(By.id("what")).clear(); Thread.sleep(2000); driver.findElement(By.id("what")).sendKeys("Java"); // // Enter the keyword, "Hyderabad" in the field, Where // driver.findElement(By.id("where")).clear(); Thread.sleep(2000); driver.findElement(By.id("where")).sendKeys("Hyderabad"); - Submit and get the results page. The following command would result in submitting the inputs to be submitted.
driver.findElement(By.id("fj")).click(); - Analyze the results; The following are sample commands:
System.out.println("Page Title:" + driver.getTitle()); System.out.println("Jobs Count: " + driver.findElement(By.id("searchCount")).getText());
- Provide the inputs. In the example given above, the input fields are cleared first, and then, data is entered into the field. Failing to do so would append the data to already existing text in the field, if any. Note that Thread.sleep() is used to pause the steps as appropriate.
- Close the browser.
driver.close();
Further Reading/References
Summary
In this post, you learned about how to use Selenium for Web Scraping using Java programming language.
Did you find this article useful? Do you have any questions or suggestions about this article? Leave a comment and ask your questions and I shall do my best to address your queries.
Check out my latest book titled as First Principles Thinking: Building winning products using first principles thinking.
- Agentic Reasoning Design Patterns in AI: Examples - October 18, 2024
- LLMs for Adaptive Learning & Personalized Education - October 8, 2024
- Sparse Mixture of Experts (MoE) Models: Examples - October 6, 2024
I found it very helpful. However the differences are not too understandable for me