Grow your CSS skills. Land your dream job.

Scraping a website

  • # June 15, 2013 at 2:54 pm

    I need to scrape a website using JS for a mobile app being built with phonegap.

    I have read that scraping using node.js is simple. So is there a way to tie up the app to node.js app that scrapes the websites and returns a json to the app. If so how? and what will I need

    # June 16, 2013 at 12:15 am

    Scraping is never an exact science, so if it’s a premium app I don’t know if you’re users will be happy with the results.

    I tested out http://pipes.yahoo.com/pipes/ for a site to display a combination of news feeds from various sites into one feed, it allows for a lot of fine targeting and control of content, so might be worth looking at depending on your needs.

    # June 16, 2013 at 1:31 am

    >If so how?

    getting JSON from a server usually involves an HTTP request (e.g., AJAX).

    >and what will I need

    You’ll need to learn how to use AJAX and node.js, for starters.

    You should also make sure the sites in question will **allow** you to scrape their content. Usually, if a site wants to allow you access to their content, they’ll provide an API for the purpose (and there will be no need to try and figure it out yourself). Remember that copyright is _implied_ in most jurisdictions, and failure to refuse is not the same as granting permission. Ask first.

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.

*May or may not contain any actual "CSS" or "Tricks".