r/scrapinghub • • Jan 02 '19

why dont we all pool our unused mobiles together

17 Upvotes

r/scrapinghub • • Mar 05 '19

Meet Spidermon: Scrapinghub’s Battle Tested Spider Monitoring Library [Now Open Sourced]

14 Upvotes

Read on how to Install Spidermon and start using TODAY to monitor your Scrapy Spiders -

https://blog.scrapinghub.com/spidermon-scrapy-spider-monitoring


r/scrapinghub • • May 14 '19

ScrapyRT: Turn Websites Into Real-Time APIs

11 Upvotes

If you’ve been using Scrapy for any period of time, you know the capabilities a well-designed Scrapy spider can give you.

With a couple lines of code you can design a scalable web crawler and extractor that will automatically navigate to your target website and extract the data you need. Be it e-commerce, article or sentiment data.

The one issue that traditional Scrapy spiders poses however, is the fact that in a lot of cases spiders can take a long time to finish their crawls and deliver their data if it is a large job. With the growth of data based services and data-driven decision making, end users are increasingly looking for ways to extract data on demand from web pages instead of having to wait for data from large periodic crawls.

And that’s where ScrapyRT comes in…

Simply send your Scrapy HTTP API a request containing the Scrapy Request Object (with URL and callback as parameters) and the API will return the extracted data by the spider in real-time. No need to wait for the entire crawl to complete.

https://blog.scrapinghub.com/scrapyrt-turn-websites-into-real-time-apis

If you would like to learn more about ScrapyRT or contribute to the open source project, then check out the ScrapyRT documentation and GitHub repository.


r/scrapinghub • • Aug 16 '21

Extract Summit 2021 is here!

10 Upvotes

Save the date: 30th September 2021!
The most awaited event in the web data extraction industry will be here in 45 days!

It's a perfect opportunity to hear from the thought-leaders in the industry and meet hundreds of like-minded web data lovers!

Check the agenda - https://www.extractsummit.io/web-data-extraction-summit-2021-agenda/

Grab your free ticket - https://www.extractsummit.io/register-for-extract-summit/


r/scrapinghub • • Nov 08 '19

Scrapy, Matplotlib and MySQL: Real Estate Data Analysis

9 Upvotes

New blog post: Scrapy, Matplotlib and MySQL: Real Estate Data Analysis

Our new blog post shows how to extract data from real-estate websites and then analyse the data.

Tools and libraries used are:

  • Scrapy for web scraping

  • MySQL to store data

  • Pandas to query and structure data in code

  • Matplotlib to visualize data

Read the full blog post here - https://blog.scrapinghub.com/scrapy-matplotlib-and-mysql-real-estate-data-analysis


r/scrapinghub • • Mar 03 '20

Building Spiders Made Easy: GUI For Your Scrapy Shell

8 Upvotes

Roy Healy, our python developer, created this amazing open-source project on Scrapy Shell GUI. Read it here - https://blog.scrapinghub.com/building-spiders-made-easy-gui-for-your-scrapy-shell


r/scrapinghub • • Nov 26 '19

Price Scraping: The Best Free Tool To Scrape Prices

8 Upvotes

New Blog Post - https://blog.scrapinghub.com/price-scraping-best-free-tool-to-scrape-prices

The use-cases of price scraping are endless. It might look easy and just a minor technical detail that needs to be handled but in reality, if you don’t know the best way to get those price values from the HTMLs, it can be a headache over time.

In this article, first, we will show you some examples where price scraping is essential for business success. We will then learn how to use our open-source python library, price-parser. This library was made specifically to make it easy to extract clean price information from an e-commerce site.

Read the full blog post to learn how to efficiently scrape prices from e-commerce websites using price-parser.


r/scrapinghub • • Mar 28 '19

How To Architect A Web Scraping Solution: The Step-By-Step Guide.

8 Upvotes

New blog post - How To Architect A Web Scraping Solution: The Step-By-Step Guide.

"For many people (especially non-techies), trying to architect a web scraping solution for their needs and estimate the resources required to develop it, can be a tricky process.

Oftentimes, this is their first web scraping project and as a result have little reference experience to draw upon when investigating the feasibility of a data extraction project.

In this series of articles we’re going to break down each step of Scrapinghub’s four step solution architecture process so you can better scope and plan your own web scraping projects......."

https://blog.scrapinghub.com/architecting-a-web-scraping-solution


r/scrapinghub • • Jul 23 '20

Want to speak at Extract Summit 2020?

7 Upvotes

Extract Summit is one of its kind event focussed on bringing together hundreds of data enthusiasts in the field of web scraping and data extraction. It’s a place for like-minded data lovers to fuel their ambition and spark their imagination.

We are inviting experts and innovators in data extraction and web scraping to share their ideas at Web Data Extraction Summit. Grab this opportunity to establish yourself as a pioneer in the industry. The application to speak at the Summit is now open.

Apply to speak now!


r/scrapinghub • • May 20 '20

PRODUCT REVIEWS API (BETA): EXTRACT PRODUCT REVIEWS AT SCALE

7 Upvotes

We are excited to announce our next AutoExtract API: Product Reviews API (Beta). Using this API, you can get access to product reviews in a structured format, without writing site-specific code. You can use the Product Reviews API to extract product reviews from eCommerce sites at scale. Just make a request to the API and receive your data in real-time!

E-commerce Product Reviews data extraction

In today’s competitive eCommerce world, product reviews provide a great way for online shoppers to determine what products to buy. Hence, monitoring product reviews are important for businesses. Making use of reviews data, you can find insights in the data that can improve your decision making, address feedback, and monitor customer sentiment.

But getting access to structured web data is not easy, especially if you don’t have the right tools. With Product Reviews API, we provide a convenient way for you to extract reviews at scale from any site.

Product Reviews data at your fingertips

Data fields that Product Reviews can extract for you:

  • Name (name of the review)

  • Review body

  • Date published

  • Review rating

  • Is verified

  • Voted helpful/unhelpful

  • URL

More info about the fields in the docs.

You can use Product Reviews API for:

  • Building a product

  • Sentiment analysis and NLP

  • Reputation monitoring

  • Market research

Whichever your use case is, you can always rely on Product Reviews API to deliver high-quality data.

Structured Product Reviews data, without coding

Before our Product Reviews API, you needed to write a site-specific code to extract reviews or other data. Furthermore, you also needed to maintain the code if the website changed its layout or frontend code.

With AutoExtract Product Reviews API, you don’t need to write custom code to extract data. Our AI-based tool will automatically find all the data fields you need and extract it from the page. You just need to submit the target page URLs. Then, you will receive your data in a structured JSON format.

How to use Product Reviews API

Product Reviews API works the same way as other AutoExtract APIs:

  1. Feed page URLs into AutoExtract.

  2. Receive data in JSON

Be aware, only the site URL is not enough to extract the data. You need specific page URLs to use the API (Or reach out to us to get URL discovery handled for you.)

For more information about the API check the Product Reviews API documentation.

Visual representation of Product Reviews API

📷

JSON example

This is the format you should expect when using the API

[ { "productReviews": { "url": "https://example.com/product-review", "reviews": [ { "name": "A great tool!", "reviewBody": "AutoExtract is a great tool for review extraction", "reviewRating": { "ratingValue": 5.0, "bestRating": 5.0 }, "datePublished": "2020-01-30T00:00:00", "datePublishedRaw": "Jan 30, 2020", "votedHelpful": 12, "votedUnhelpful": 1, "isVerified": true, "probability": 0.95 }, { "name": "Another review", "probability": 0.95 } ] }, "query": { "id": "1564747029122-9e02a1868d70b7a3", "domain": "example.com", "userQuery": { "pageType": "productReviews", "url": "https://example.com/product-review" } } } ]

Read more about the fields in the docs.

Try the Product Reviews Beta API Today!

Here’s what you need to do if you want to get access to the AutoExtract Product Reviews API beta:

  1. Sign up for a free trial here.

  2. You can start using the Product Reviews API straight away.

Product Reviews API is free for 14 days or until you reach 10K requests (whichever comes sooner). After that, you will be billed $60/month if you don’t cancel your subscription.

If you want to try the Product Reviews API Beta, sign up here for free!


r/scrapinghub • • Apr 21 '20

Looking for beta testers!

7 Upvotes

Hi community,

I’ve been working on BitPull, a web scraping service that started out as a side project to make data extraction easy.

The cool thing about it is that you don’t need any coding knowledge, you can just create modular workflows to satisfy your data scraping needs.

I’m looking for beta testers willing to use it in some real world scenarios. You can just sign up for free and start using it right now.

Here’s some of its features:

  • Point and click data you need

  • Test your workflow straight away, and see results immediately

  • Export to services like Google Drive, OneDrive, Github and Dropbox

  • Pagination, login and much more

  • Export to pdf, take a screenshot or write to excel

  • Job scheduling

  • Get automatically notified on Slack or by email

  • Data previews

If this is something you like, I’d love for you to check it out and provide some feedback!

You can check it out here: https://bitpull.io

There are a bunch of examples to start with, including a profile scraper for LinkedIn.

Thanks,

Frederik


r/scrapinghub • • Apr 10 '19

Scrapinghub's New AI Powered Developer Data Extraction API for E-Commerce & Article Extraction

7 Upvotes

New blog post: Scrapinghub's New AI Powered Developer Data Extraction API for E-Commerce & Article Extraction

Today, we’re delighted to announce the launch of the beta program for Scrapinghub’s new AI powered developer data extraction API for automated product and article extraction.

After much development and refinement with alpha users, our team have refined this machine learning technology to the point that data extraction engine is capable of automatically identifying common items on product and article web pages and extracting them without the need to develop and maintain individual web crawlers for each site.

Enabling developers to easily turn unstructured product and article pages into structured datasets at a scale, speed and flexibility that is nearly impossible to achieve when manually developing spiders.

With the AI enabled data extraction engine contained within the developer API, you now have the potential to extract product data from 100,000 e-commerce sites without having to write 100,000 custom spiders for each. 

As result, today we’re delighted to announce the launch of the developer API's public beta.

Join The Beta Program Today! - https://scrapinghub.com/developer-api


r/scrapinghub • • Dec 26 '20

How to learn everything or at least most important things about developer tools of a browser

6 Upvotes

In my previous web scraping project i saw some amazing things a developer tool of any browser can do but i was wondering where can i learn or get more tips and tricks you can do with this tool?


r/scrapinghub • • Sep 21 '20

3 Most Practical Uses of eCommerce Data Scraping Tools NSFW Spoiler

6 Upvotes

Table of Contents

3 Practical Uses of Ecommerce Data

3 popular eCommerce data scraping tools

Conclusion

In today’s eCommerce world, eCommerce data scraping tools gain great popularity all over the world as the competition among eCommerce business owners gets more fierce year by year. Data scraping tools become the new technique and tool to help them improve their performance.

A lot of store owners find using an eCommerce data scraping tool to monitor competitors’ activities and customers’ behaviors can help them maintain their competitiveness and improve sales. If you have no idea how to make full use of eCommerce data scraping tools, stay with me and we will look into 3 most practical uses of a scraping tool and how the tool helps grow your business. 

Three Practical Uses of Ecommerce Data

1) Price Monitoring

Price is one of the most critical aspects that affect customers’ buying interest. 87% of online shoppers indicate that price is the most important factor that affects buying behaviors, followed by shipping cost and speed. That research suggests that a potential customer won’t hesitate to leave your store if your price doesn’t match his expectation.

In addition, according to a study from AYTM, 78 percent of shoppers compare prices between two or more brands, then opt for the lowest price. With easy access to many free online price comparison tools, online shoppers can easily see the price of a specific item across dozens of brands and marketplaces.

It is necessary for online business owners to have an eCommerce data scraping tool to scrape price information from competitors’ web pages or from price comparison Apps. If not, it’s likely that you will have trouble attracting new customers to your store or maintaining your current customer base, because you don’t know when and how to adjust your price to cater to those price-sensitive customers.

📷

2) Competitor Analysis

We’re aware that improving the shipping service is another solution to increase sales. 56% online sellers offer free shipping (and easy returns) regardless of the purchase price or the product type. 

Lots of online sellers use free shipping as a marketing strategy to nudge people to buy from them or even buy more from them. For example,  it’s quite common that customers are more willing to spend $100 on a product with free shipping rather than buy a $90 product that takes $10 for the shipping. Besides, it’s common for customers to buy more items in order to get a free shipping offer.

You can use an eCommerce data scraping tool to find out how many of your competitors are offering a free shipping service. Using a data scraping tool, you can easily scrape and collect the data in real-time. In this case, if they don’t provide a free shipping service, you can attract their customers by offering it.

3) Customer Sentiment Analysis

Knowing how your competitors’ audiences feel about the products or brands can help you evaluate your marketing strategy and customer experience management. ECommerce data scraping tools can help you gather such information.

The voices of customers that you gather from your competitors will help you understand what customers value and how you can better serve them. Their voices are mostly scattered among comments and conversation under your competitors’ stores and posts and interactions on their social media. With such information at hand, you will know what customers want from the product – what they like or dislike. 

To outcompete your competitors, it is necessary for you to gain all those information, look into it and draw conclusions. Therefore you can adjust your marketing strategy or your products/services accordingly.

📷

Now you are probably wondering what scraping tools can be used for these purposes. Here, I would like to share with you this shortlist of the most popular eCommerce data scraping tools. You should try them out!

3 popular eCommerce data scraping tools

1) Octoparse

Octoparse is a free and powerful eCommerce data scraping tool with a user-friendly point-and-click interface. Both Windows and Mac users will find it easy-to-use for extracting almost all kinds of data you need from a website. With its brand new auto-detect algorithm, users with/without coding knowledge are able to extract tons of data within seconds.  

Pros: Octoparse provides over 50 pre-built templates for all users, covering big websites such as Amazon, Facebook, Twitter, Instagram, Walmart, etc. All you need to do is to enter the keywords and URL, then wait for the data result. In addition, it provides a free version for all people. For premium users, they can use features such as crawler scheduling and cloud extraction to make the process less time-consuming. 

Cons: Octoparse cannot scrape data from PDF files. It can’t download files automatically, while it allows you to extract the URLs of images, PDFs and other types of files. You can use automatic download software to down these files in bulk with the URL scraped by Octoparse.

2) Parsehub

ParseHub works with single-page apps, multi-page apps and other modern web technology. ParseHub can handle Javascript, AJAX, cookies, sessions, and redirects. You can easily fill in forms, loop through dropdowns, login to websites, click on interactive maps and deal with websites that apply infinite scrolling techniques. 

Pros: Parsehub supports both Windows and Mac OS systems. It provides a free version for people with eCommerce data scraping needs. 

Cons: The free version is quite limited with only 5 projects and 200 pages per run. It didn’t support documentation extraction. And some advanced functions are tricky to use sometimes. 

3) 80legs

80legs is a web data extraction tool that allows users to create and run web crawlers through its software as a service platform. It’s built on top of a distributed grid computing network. This grid consists of approximately 50,000 individual computers distributed across the world and uses bandwidth monitoring technology to prevent bandwidth cap overages.

Pros: 80legs is more suitable for small companies and individuals. It offers unique service plans so that customers pay only for what they crawl.

Cons: 80legs is not able to help to get a huge amount of data, you must choose between custom set crawled data, pre-built API, and crawl application to be developed. 

Conclusion

Once you know how to use eCommerce data scraping tools to help you get the needed data, what insights you can gain from the data is another story. Try to do some data analysis and find ways to visualize the data. Put your data into use.

You can try the simple analysis methods mentioned in this article to get to know your users through data analysis.


r/scrapinghub • • Apr 10 '20

A Practical Guide to Web Data Extraction QA: Common Validation Pitfalls

6 Upvotes

New Blog Post: https://blog.scrapinghub.com/web-data-qa-common-validation-pitfalls

This second blog post in our series of Practical Guides to Quality Assurance talks about the common Pitfalls that Quality Assurance engineers fall into while extracting web data and how to deal with them.

In case you missed the first part of this series, where we went through data validation techniques, you can read it now: A Practical Guide To Web Data Qa Part I: Validation Techniques


r/scrapinghub • • Jul 26 '19

GDPR Update: Scraping Public Personal Data

6 Upvotes

New Blog Post: GDPR Update: Scraping Public Personal Data

One common misconception about scraping personal data is that public personal data does not fall under the GDPR. Many businesses assume that because the data has already been made public on another website that it is fair game to scrape. Read this blog post to find out when you can or cannot scrape public personal data.

https://blog.scrapinghub.com/gdpr-public-personal-data-update?hs_preview=LMuWFYyO-11584668230


r/scrapinghub • • Apr 11 '19

AI powered Data Extraction API from the creators of Scrapy

6 Upvotes

New blog post: From The Creators Of Scrapy: Artificial Intelligence Data Extraction API

To accurately extract data from a web page, developers usually need to develop custom code for each website. This is manageable and recommended for tens or hundreds of websites and where data quality is of the utmost importance, but if you need to extract data from thousands of sites, or rapidly extract data from sites that are not yet covered by pre-existing code, this is often an insurmountable challenge.

The complex and resource intensive nature of developing code for each individual website, acts as a bottleneck severely curtailing the scope of companies data extraction and analysis capabilities.

Learn how Scrapinghub, the creators of Scrapy, have developed an AI enabled data extraction engine to enable companies to extract data from thousands of websites without having to write or maintain code.

If you are interested in large-scale product and article data extraction and would like to get early access to the data extraction developer API then sure to sign up today as places are limited.

https://blog.scrapinghub.com/artificial-intelligence-data-extraction-api


r/scrapinghub • • Mar 15 '19

St Patrick’s Day Special: Finding Dublin’s Best Pint of Guinness With Web Scraping

6 Upvotes

Need to Find the Best Guinness in Dublin? Web Scraping To The Rescue

https://blog.scrapinghub.com/web-scraping-best-pint-of-guinness-dublin


r/scrapinghub • • Dec 17 '18

Up to date LinkedIn scraper

6 Upvotes

Hi all, I couldn't find any up-to-date tools on GitHub to scrape LinkedIn profiles. So I built one myself

https://github.com/hakimkhalafi/linkedin-scraper

Input: Search string for type of profile eg. "data scientist"

Output: CSV file with Top X people matching that search query and their full profile information (workplace, education, etc)

Feel free to use it however you see fit. It spits out a nice CSV which can easily be analysed as a DataFrame with Text Analytics tools.


r/scrapinghub • • Aug 10 '22

Extract Summit 2022 is back in-person!

5 Upvotes

Extract Summit 2022 is back in-person! It's going to be on 29th September in London!

Extract Summit is an event dedicated to web data extraction. Thought leaders from various industries gather to talk about the innovations and trends in web scraping. The in-person event will bring lots of opportunities for networking.

This year, a lot of the talks are dedicated to web scraping best practices and how to get the best quality data with the least possible obstacles.

Check out the full agenda here - https://www.extractsummit.io/agenda/
Meet the speakers for 2022 - https://www.extractsummit.io/#speakers


r/scrapinghub • • May 21 '21

Scraped FB Group users, now what?

5 Upvotes

So, used PhantomBuster to scrape a group where I could find potential leads. But now what do I do with the list?

How to get them to like my Fan Page or join another FB group, or even visit my website to be tag by FB Pixel.

I'm a bit lost at this point what would you do to use this list of users? Which other tools could I use to achieve that?

Thanks!


r/scrapinghub • • May 14 '21

Want to speak at Extract Summit 2021?

5 Upvotes

The Extract Summit season has officially begun!
Extract Summit is a single platform for all data lovers to come together to educate, inspire, and innovate.

If you have a story that will inspire thousands of web data lovers, we want to hear from you! Apply to speak at Extract Summit - https://www.extractsummit.io/speak/


r/scrapinghub • • Sep 08 '20

The Web Data Extraction Summit 2020

5 Upvotes

We are delighted to announce that Scrapinghub will be once again hosting the Web Data Extraction Summit this year on Tuesday, November 10th, 2020.

Extract Summit 2020 is going to be a completely free-to-attend and virtual event making it accessible for data enthusiasts all over the world to network and learn from each other. All you need is a laptop or a phone to get instant access to lots of amazing talks and connect with hundreds of other data lovers like you.

Register for Free!


r/scrapinghub • • Sep 03 '20

Is scraping a website and using its content on another website legal ?

5 Upvotes

I am developing a website and I thought about scraping the content of other websites and displaying it on my website, will I get in trouble for doing this ?


r/scrapinghub • • Feb 19 '20

Introducing Crawlera Free Trials & New Plans!

5 Upvotes

You can now try the world's smartest proxy network - Crawlera for free!

We have also introduced a new set of plans based on the feedback received from our customers. Read the blog post for more details! Experience the Crawlera performance and reliability through the Free Trial now! - https://blog.scrapinghub.com/introducing-crawlera-free-trials-new-plans