
Code Sample – Get Content from URL
Pay attention to some of the following aspects of fetching content from a given URL:
- Create a URL object using actual URL string
- Create a URLConnection object using that URL object created in above step
- Set the configuration parameters. Key is to note the connection and read timeout. At times when scraping the websites, it helps with slow websites.
- Create a BufferedReader for reading the data
- Read line by line
public String getContent(String urlstr) { URL url = null; StringBuilder contentb = new StringBuilder(); try { // get URL content url = new URL(urlstr); // Create a URL Connection Object URLConnection conn = (HttpURLConnection) url.openConnection(); // Set the configuration parameters // Note the readTimeOut set to 30 seconds. // This is quite important when you are planning to scrape URLs. conn.setConnectTimeout(100000); conn.setReadTimeout(30000); conn.connect(); // open the stream and put it into BufferedReader BufferedReader br = new BufferedReader(new InputStreamReader( conn.getInputStream())); String inputLine; while ((inputLine = br.readLine()) != null) { contentb.append(inputLine); contentb.append("\n"); } br.close(); } catch (MalformedURLException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } return contentb.toString(); }
- Climate Analysis & Top Questions for Leadership - January 5, 2023
- Types of SQL Joins Explained with Examples - January 3, 2023
- Types of Frequency Distribution & Examples - January 2, 2023
Leave a Reply