English | MP4 | AVC 1280×720 | AAC 44KHz 2ch | 8 Hours | 5.12 GB
Learn web scraping in Nodejs by example projects with real websites! Craiglist, iMDB, AirBnB and more!
In this course you will learn how to scrape a websites, with practical examples on real websites using Nodejs Request, Cheerio, NightmareJs and Puppeteer. You will be using the newest JavaScript ES7 syntax with async/await.
You will learn how to scrape a Craigslist website for software engineering jobs, using Nodejs Request and Cheerio. You will be using the newest JavaScript ES7 syntax with async/await.
You will then learn how to scrape more advanced websites that require JavaScript such as iMDB and AirBnB using NighmareJs and Puppeteer.
I’m gong to also show you with a practical real-life website, how you can even avoid wasting time on creating a web scraper in the first place, by reverse engineering websites and finding their hidden API’s!
Learn how to avoid being blocked from websites when developing out your scraper, by building out the scraper in a test-driven way with mocked html, rather than hitting the website every time as you’re debugging and developing it. You’ll also learn what you can do if you’re blocked and your alternatives to get your scraper up and running regardless!
You will also learn how to scrape on a server with a bad connection, or even if you have a bad connection.
You’ll even learn how to save your results to a CSV file and MongoDB!
How do you build a scraper that scrapes every 1 hour (or other interval), and deploy it do a cloud host like Heroku or Google Cloud? Let me show you, quick and easy!
How do you scrape a site requiring passwords? I’m going to show you that too with a real website (Craigslist)!
How do you serve your scraping results in a REST API with Nodejs Express? And how can we build a React frontend that’s showing the results? You’ll learn that too, in the quickest and simplest way possible!
Plus, a section covering how to make a basic GraphQL API is included in the course.
As a last cherry on the top, I have a section containing a secret backdoor showing you how to scrape Facebook using only Request!
What you’ll learn
- Be able to scrape jobs from a page on Craigslist
- Learn how to use Request
- Learn how to use NightmareJS
- Learn how to use Puppeteer
- Learn how to scrape elements without any identifiable classes or id’s
- Learn how to save scraping data to CSV
- Learn how to save scraping data to MongoDb
- Learn how to scrape Facebook using only Request!
- Learn how you can reverse engineer sites and find hidden API’s!
- Learn different technologies used for scraping, and when it’s best to use them
- Learn how to scrape sites using authentication
Table of Contents
Required Software
Software to web scrape in JavaScript
What you should ALWAYS check before even writing a web scraper!
This could save you A LOT of time and effort!
Intro to CSS selectors and tools we use for scraping
Intro to section
Using Chrome Developer Tools
Selecting our element
Building our first scraper!
Selecting multiple elements
Selecting using CSS ID
Selecting using CSS classes
Selecting using HTML attributes
You’re on your way to become a scraping ninja!
Web Scraping Craigslist Jobs
Introduction
Outtro
Project Setup
Getting Html from website
Creating sample of data to collect
TitleURL From Jobs
Scraping Time Job Was Posted
Job Neighborhood
Scraping Job Descriptions
Finish Description and Compensation
Exporting web scraping results to CSV
Exporting web scraping results to CSV
Handling Network Problems
Handling Network Problems in our Craigslist scraper
Scraping Sites with Pagination
Simple Pagination Scraper in 10 mins!
Scraping Sites with Authentication
Intro to authentication scraping project
Looking at Login request
Recreating login in Postman
Creating our login request in Nodejs
Using Puppeteer instead of Request
Scraping Nordstrom.com – how to find a secret API and avoid building a scraper!
Intro To Nordstrom.com project
Adding search query to form
Finding the secret API behind Nordstrom.com
Making a API request inside Postman
Creating a REST API in Nodejs Express
Passing Query Parameters to our own REST API
Starting React project with create-react-app
Making a API Request inside the React app
Something
Adding a form to React app
Scraping Imdb using NightmareJs
Intro to Project
Importing NightmareJs and Getting Our Poster Image Css Selector
Scraping the Poster Image URL with NightmareJs
Saving the Poster Image to Disk!
Project Setup
So What Are We Scraping
Scraping Top 100 Movie Titles
Let’s Get Some Good Ratings!
Easy Peasy Rank and Description Url
Css Selector For The Poster Url
Scraping The Poster URL
Why Request Can’t Scrape This Page – Why We’re Using NightmareJs Now
Scraping AirBnb using Puppeteer
Intro to Project
Scraping Price Per Night
Why we are using Regular Expressions now
Scraping number of guests allowed using regular expressions
Scraping the beds, bedrooms, baths
Project Setup
What are we scraping exactly
Sample Object + Index Offset
Looking at the HTML of the Index page
Opening Page with Puppeteer
Getting the URLS of the homes from the index page
Getting ready to scrape description URLS
Opening Homes in a separate page
Architecture for web scraper with an API
Intro to this section
Timed scraping vs on-demand scraping API’s
Saving scraping data to MongoDB
Build a super simple Reddit scraper in 5 minutes
Connecting to MongoDB database
Connecting to MongoDB database using Mongoose
Creating a MongoDB model and saving
Deploying a periodic scraper to production
Intro
Intro to code
Deploying to Heroku
Deploying to Google Cloud Platform Google App Engine
BONUS GraphQL Introduction
Introduction to GraphQL + Creating a GraphQL API in 10 minutes
Scraping a infinite scrolling page (Facebook, Instagram, Pinterest etc.)
Intro to scraping infinite scrolling pages
Project setup
Extracting items function
Scrolling and Scraping items
SECRET BACKDOOR to Scraping Facebook without JavaScript enabled!
How to get access to Facebook’s site without JavaScript
How to use Postman to get Facebook’s wall
Project Setup for Facebook Scraper
Creating our POST request in Nodejs
Faking our User-Agent and logging in to Facebook
Getting our Facebook wall!
BONUS – Coupons for other courses
Coupons for my other courses
Resolve the captcha to access the links!