The Lodge is members-only design/dev videos and Office Hours.

Next Office Hours Session: "Implementing an SVG Icon System" Nov 30 - 6:00 PM Eastern

Scraping a website

  • # June 15, 2013 at 2:54 pm

    I need to scrape a website using JS for a mobile app being built with phonegap.

    I have read that scraping using node.js is simple. So is there a way to tie up the app to node.js app that scrapes the websites and returns a json to the app. If so how? and what will I need

    # June 16, 2013 at 12:15 am

    This reply has been reported for inappropriate content.

    Scraping is never an exact science, so if it’s a premium app I don’t know if you’re users will be happy with the results.

    I tested out for a site to display a combination of news feeds from various sites into one feed, it allows for a lot of fine targeting and control of content, so might be worth looking at depending on your needs.

    # June 16, 2013 at 1:31 am

    >If so how?

    getting JSON from a server usually involves an HTTP request (e.g., AJAX).

    >and what will I need

    You’ll need to learn how to use AJAX and node.js, for starters.

    You should also make sure the sites in question will **allow** you to scrape their content. Usually, if a site wants to allow you access to their content, they’ll provide an API for the purpose (and there will be no need to try and figure it out yourself). Remember that copyright is _implied_ in most jurisdictions, and failure to refuse is not the same as granting permission. Ask first.

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.

There's a whole bunch of content on CSS-Tricks.

Search for Stuff   •   Browse the Archives

Get the Newsletter ... or get the RSS feed