Java – How to Scrape Content from a URL

This article represents take-away code sample that could be used to get or scrape content from a given URL. Those wanting to do a quick web scrape could use this piece of code. I shall be posting a series of blogs which would help one to create a web scraper using Java. The reason why I am hooked to Java web scraping is the need to get the data from web for data analysis. Please feel free to comment/suggest if I missed to mention one or more important points. Also, sorry for the typos.

Code Sample – Get Content from URL

Pay attention to some of the following aspects of fetching content from a given URL:

• Create a URL object using actual URL string
• Create a URLConnection object using that URL object created in above step
• Set the configuration parameters. Key is to note the connection and read timeout. At times when scraping the websites, it helps with slow websites.
• Create a BufferedReader for reading the data
• Read line by line

	public String getContent(String urlstr) {
		URL url = null;
		StringBuilder contentb = new StringBuilder();
		try {
			// get URL content
			url = new URL(urlstr);
			// Create a URL Connection Object
			URLConnection conn = (HttpURLConnection) url.openConnection();
			// Set the configuration parameters
			// Note the readTimeOut set to 30 seconds.
			// This is quite important when you are planning to scrape URLs. 
			conn.setConnectTimeout(100000);
			conn.setReadTimeout(30000);
			conn.connect();
			// open the stream and put it into BufferedReader
			BufferedReader br = new BufferedReader(new InputStreamReader(
					conn.getInputStream()));
			String inputLine;
			while ((inputLine = br.readLine()) != null) {
				contentb.append(inputLine);
				contentb.append("\n");
			}
			br.close();
		} catch (MalformedURLException e) {
			e.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		}
		return contentb.toString();
	}