Selenium is, primarily, used for automating web applications for testing purposes. However, it could also be used for doing web scraping if required. In this post, you would learn about how to use Selenium for Web Scraping using Java. The following are some of the topics described in this article.
Create a Maven Project within your Eclipse IDE and place the following entry in pom.xml file. Get the latest artifacts from Maven Page for Selenium Java page.
<!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java --> <dependency> <groupId>org.seleniumhq.selenium</groupId> <artifactId>selenium-java</artifactId> <version>3.11.0</version> </dependency>
The above would result in the download of appropriate Java libraries (JAR files) to run Selenium program. The following screenshot represents the downloads and where to find them.
The following is the sample code for feeding the inputs to Indeed home page (https://www.indeed.co.in) for searching jobs in a particular location. The code given below automates the job search by providing inputs for “What” and “Where” field and clicking on the “Find Jobs” button.
import java.io.File; import org.openqa.selenium.By; import org.openqa.selenium.WebDriver; import org.openqa.selenium.chrome.ChromeDriver; /** * Indeed Job Search Hello World Program * */public class IndeedJobSearch { public static void main( String[] args ) throws InterruptedException { // // Set the path of the driver to driver executable. For Chrome, set the properties as following: // File file = new File("D:/softwares/selenium/chromedriver_win32/chromedriver.exe"); System.setProperty("webdriver.chrome.driver", file.getAbsolutePath()); // Create a Chrome Web Driver // WebDriver driver = new ChromeDriver(); // // Open the Indeed.com homepage // driver.get("https://www.indeed.co.in"); // // Enter the keyword "Java" in the field, What // driver.findElement(By.id("what")).clear(); Thread.sleep(2000); driver.findElement(By.id("what")).sendKeys("Java"); // // Enter the keyword, "Hyderabad" in the field, Where // driver.findElement(By.id("where")).clear(); Thread.sleep(2000); driver.findElement(By.id("where")).sendKeys("Hyderabad"); // // Click the FindJobs button for searching // driver.findElement(By.id("fj")).click(); // // Print the information from the new page // System.out.println("Page Title:" + driver.getTitle()); System.out.println("Jobs Count: " + driver.findElement(By.id("searchCount")).getText()); Thread.sleep(2000); // // Close the browser // driver.close(); } }
Pay attention to some of the following in the code given above.
File file = new File("D:/softwares/selenium/chromedriver_win32/chromedriver.exe"); System.setProperty("webdriver.chrome.driver", file.getAbsolutePath());
WebDriver driver = new ChromeDriver();
driver.get("https://www.indeed.co.in");
// // Enter the keyword "Java" in the field, What // driver.findElement(By.id("what")).clear(); Thread.sleep(2000); driver.findElement(By.id("what")).sendKeys("Java"); // // Enter the keyword, "Hyderabad" in the field, Where // driver.findElement(By.id("where")).clear(); Thread.sleep(2000); driver.findElement(By.id("where")).sendKeys("Hyderabad");
driver.findElement(By.id("fj")).click();
System.out.println("Page Title:" + driver.getTitle()); System.out.println("Jobs Count: " + driver.findElement(By.id("searchCount")).getText());
driver.close();
In this post, you learned about how to use Selenium for Web Scraping using Java programming language.
Did you find this article useful? Do you have any questions or suggestions about this article? Leave a comment and ask your questions and I shall do my best to address your queries.
In recent years, artificial intelligence (AI) has evolved to include more sophisticated and capable agents,…
Adaptive learning helps in tailoring learning experiences to fit the unique needs of each student.…
With the increasing demand for more powerful machine learning (ML) systems that can handle diverse…
Anxiety is a common mental health condition that affects millions of people around the world.…
In machine learning, confounder features or variables can significantly affect the accuracy and validity of…
Last updated: 26 Sept, 2024 Credit card fraud detection is a major concern for credit…