r/scrapinghub • • Jun 21 '19

VISUAL WEB SCRAPING TOOLS: WHAT TO DO WHEN THEY ARE NO LONGER FIT FOR PURPOSE?

0 Upvotes

New Blog Post: VISUAL WEB SCRAPING TOOLS: WHAT TO DO WHEN THEY ARE NO LONGER FIT FOR PURPOSE?

Visual web scraping tools are great. They allow people with little to no technical know-how to extract data from websites with only a couple hours of upskilling, making them great for simple lead generation, market intelligence and competitor monitoring projects. Removing countless hours of manual entry work for sales and marketing teams, researchers, and business intelligence team in the process.

However, no matter how sophisticated the creators of these tools say their visual web scraping tools are, users often run into issues when trying to scrape mission-critical data from complex websites or when scraping the web at scale.

In this article, we’re going to talk about the biggest issues companies face when using visual web scraping tools like Mozenda, Import.io and Dexi.io, and what they should do when they are no longer fit for purpose.

https://blog.scrapinghub.com/visual-web-scraping-tools-what-to-do-when-they-are-no-longer-fit-for-purpose


r/scrapinghub • • Sep 07 '20

Web Data Extraction: The Definitive Guide 2020 NSFW Spoiler

0 Upvotes

Web data extraction is gaining popularity as one of the great ways to collect useful data to fuel the business cost-effectively. Although web data extraction has existed for quite some time, it has never been as heavily used, or as reliable as it is today. This guide aims to help web scraping beginners to get a general idea of web data extraction.

Table of Contents

   What is web data extraction

+ Benefits of web data extraction

  • E-commerce price monitoring

  • Marketing analysis

  • Lead generation

+ Web data extraction for non-programmers

  • Octoparse

  • Cyotek WebCopy

  • Getleft

  • OutWit Hub

  • WebHarvy

   Conclusions

What is web data extraction

Web data extraction is a practice of massive data copying done by bots. It has many names, depending on how people would like to call it, web scraping, data scraping, web crawling, to name a few. The data extracted(copied) from the internet can be saved to a file in your computer, or database.

Benefits of web data extraction

Businesses can get a load of benefits from web data extraction. It can be used more widely than you expect, but it would suffice to point out how it is used in a few areas.

1 E-commerce price monitoring

The importance of price monitoring speaks for itself, especially when you sell items on an online marketplace such as Amazon, eBay, Lazada, etc. These platforms are transparent, that is, buyers, also any one of your competitors, have easy access to prices, inventory, reviews, and all kinds of information for each store. which means you can’t just focus on the price but also need to keep an eye on other aspects of your competitors. Hence in addition to prices, there are more available for you to dig into. Price monitoring may be more than prices.

Most retailers and e-commerce vendors try to put as much information about their products online as possible. This is helpful for buyers to evaluate, but also is too much exposure for the store owners because, with such information, competitors can get a glimpse of how you run your business. Fortunately, you can use these data to do the same thing.

You should gather information such as price, inventory levels, discounts, product turnover, new items added, new locations added, product category ASP, etc, from your competitors as well. With these data at hand, you can fuel your business with below benefits rendered by web data extraction.

  1. Increase margins and sales by adjusting prices at the right time on the right channels.

  2. Maintain or improve your competitiveness in the marketplace.

  3. Improve your cost management by using competitor prices as a negotiating ground with suppliers, or review your own overheads and production cost.

  4. Come up with effective pricing strategies, especially during promotion such as season-end sales or holiday seasons.

2 Marketing Analysis

Almost everyone can start their own business as long as they go online thanks to the easy entry brought by the magic Internet. Businesses increasingly sprout on the Internet signifies that competition among retailers will be more fierce. To make your business stand out and to maintain sustainable growth, you can do more than just lower your price or launch advertising campaigns. They could be productive for a business in an initial stage, while in the long run, you should keep an eye on what other players are doing and condition your strategies to the ever-changing environment.

You can study your customers and your competitors by scraping product prices, customer behaviors, product reviews, events, stock levels, and demands, etc. With this information, you’ll gain insights on how to improve your service and products and how to stand out among your competitors. Web data extraction tools can streamline this process, providing you with always up-to-date information for marketing analysis.

Get a better understanding of your customers’ demands and behaviors, and then find some specific customers’ needs to make exclusive offerings.

  1. Analyze customer reviews and feedback for products and services of your competitors to make improvements to your own product.

  2. Make a predictive analysis to help foresee future trends, plan future strategies and timely optimize your prioritization.

  3. Study your competitors’ copies and product images to find out the most suitable ways to differentiate yourself.

3 Lead generation

There is no doubt that being capable of generating more leads is one of the significant skills to grow your business. How to generate leads effectively? A lot of people talk about it but few of them know how to make it. Most salespeople, however, are still looking for leads on the Internet in a traditional, manual way. What a typical example of wasting time on trivia.

Nowadays, smart salespeople will search for leads with the help of web scraping tools, running through social media, online directories, websites, forums, etc, so as to save more time to work on their promising clients. Just leave this meaningless and boring lead copying work to your crawlers.

When you use a web crawler, don’t forget to collect the information below for lead analysis. After all, not every lead is worth spending time on. You need to prioritize the prospects who are ready or willing to buy from you.

  1. Personal information: Name, age, education, phone number, job position, email

  2. Company information:  Industry, size, website, location, profitability

As time passes by, you’ll collect a lot of leads, even enough to build your own CRM. Having a database of email addresses of your target audience, you can send out information, newsletters, invitations for an event or advertisement campaigns in bulk. But beware of being too spammy!

How does web data extraction work?

After knowing what you can benefit from a web data extraction tool, you may want to build one on your own to harvest the fruits of this technique. It’s important to first understand how a crawler works and what web pages are built on before starting your journey of web data extraction.

  1. Build a crawler with programming languages and then enter the URL of a website that you want to scrape from. It sends an HTTP request to the URL of the webpage. If the site grants you access, it responds to your request by returning the content of webpages.

  2. Parse the webpage is only half of the web scraping. The scraper inspects the page and interprets a tree structure of the HTML. The tree structure works as a navigator will help the crawler follow the paths through the web structure to get the data.

  3. After that, the web data extraction tool extracts the data fields you require to scrape and store it. Lastly, when the extraction is finished, choose a format and export the data scraped.

The process of web scraping is easy to understand, but it’s definitely not easy to build one from scratch for non-technical people. Luckily, there are many free web data extraction tools out there thanks to the development of big data. Stay tuned, there are some nice and free scrapers I would love to recommend to you.

Web data extraction for non-programmers

Here are 5 popular web data extraction tools rated by many non-technical users. If you’re new to the web data extraction, you should give it a try.

Octoparse

Octoparse is a powerful website data extraction tool Its user-friendly point-and-click interface can guide you through the entire extraction process effortlessly. What's more, the auto-detection process and ready-to-use templates make scraping much easier for new starters.

Cyotek WebCopy

It is self-evident that WebCopy serves as a data extraction tool for websites. It is a free tool for copying full or partial websites locally onto your hard disk for offline reach. WebCopy will scan the specified website and download its content onto your hard disk. Links to resources such as style-sheets, images, and other pages on the website will automatically be remapped to match the local path. Using its extensive configuration you can define which parts of a website will be copied and how.

Getleft

Getleft is a Web-site data extraction tool. You can give it a URL, it will download a complete site according to the options specified by the user. It also changes the original pages and all the links to relative links so you can surf on your hard disk.

OutWit Hub

OutWit Hub is a Web data extraction software application designed to automatically extract information from online or local resources. It recognizes and grabs links, images, documents, contacts, recurring vocabulary and phrases, RSS feeds and converts structured and unstructured data into formatted tables which can be exported to spreadsheets or databases.

WebHarvy

WebHarvy is a point-and-click web data extraction software. It helps users easily extract data from websites to their computers. No programming/scripting knowledge is required.

Legal aspects of web data extraction

Is it legal to use a web data extraction tool? The answer depends on how you plan to use the data and whether you follow the terms of use of the website. In other words, use it within the laws.

There are a few common examples of legal and illegal activities using web scraping tools.

 Things you’re allowed to do:

  1. Use automated tools like web data extraction tools.

  2. Get access to websites like social media, e-commerce platforms, and directories to gather information.

  3. Re-publish gathered public information.

Things you’re not allowed to do:

  1. Induce harm to third-party web users (eg. posting spam comments)

  2. Induce harm to a target site functionality (eg. throttle bandwidth)

  3. Criminal activity (eg. reselling or republishing proprietary information property)

  4. Tortious conduct (eg. using that extracted info in a misleading or harmful way)

In addition, users of web data extraction tools or techniques mustn’t violate the terms of use, laws of regulations, and the copyright statements of the websites. The website will state clearly what kind of data can be used and how you can access it. You can find this information easily on its home page.

Conclusion

By now, you’ve known how powerful web data extraction can be, how it works and where you can find web data extraction tools for non-programmers. The next thing you should do is to download a tool or write a crawler to start your web crawling journey.

Regardless of what tools or techniques you are going to use to extract web data, they serve to the same end: Get helpful data to fuel your business.


r/scrapinghub • • Jun 01 '20

How can I scrap my own data off this site?

0 Upvotes

I'm pretty much done here, with the exception of one of the subreddits. Even so, I'd like to save what I've written in the past. Thanks very much for the help!


r/scrapinghub • • Sep 16 '19

Looking for something to record likes on multiple(200ish) channels/accounts

0 Upvotes

r/scrapinghub • • May 31 '21

Can I use my laptop while it is scraping the web?

0 Upvotes

Hi as the title says, can I use other software while my pc is scraping with python without stopping it?

I use bs4... ask for any information you need to determine an answer. Thanks


r/scrapinghub • • Apr 30 '20

How to scrap Indie hacker via Octoparse

0 Upvotes

Hey everyone,

New to this. Want to scrap a list of company websites I have from Indie Hacker, but the problem is Octoparse shows me the in b/w loading screen which has a random quote, instead of the actual page.

Does anyone know how to fix this?

Thanks


r/scrapinghub • • Mar 16 '20

Scraping LinkedIn Images to 800x800 ADVICE NEEDED

0 Upvotes

Hey everyone,

My scraper and I were wondering if anyone knows how to decode the token URL for LinkedIn images to the 800x800?

Anyone know how to get these URLs?


r/scrapinghub • • Mar 17 '18

Scraping Hi-Res images, Getting them Hosted, and Providing new Image URL

0 Upvotes

Hi All,

I have about 5000 website pages that need the Hi-Res Image URL scraped from.

I need them uploaded to a new host, and to have all the new URL Image links returned via Excel.

Please PM if interested.


r/scrapinghub • • Feb 19 '20

LinkedIn Scrapping

0 Upvotes

I have a csv list of 1000 people.

I wanna know for each person: if they are first connection (= I can contact them directly), or second connection i.e. I need to go thru someone in my linkedin contacts to reach out to them (And I wanna this list of people in my network that can introduce me).

Any suggestions? I keep googling and post my findings here.

I can use python. Only free stuff please.


r/scrapinghub • • Aug 30 '17

web scraping services - Stellans Technosoft

Thumbnail
stellanstechnosoft.com
0 Upvotes

r/scrapinghub • • Mar 04 '21

Which Python Library is best for Web Scraping?(Selenium, Scrapy, BeautifulSoup etc )

2 Upvotes

Hi guys,
I would like you guys to share your views on this, i am recently learning scraping, i did web scraping with BeautifulSoup and it was fun but then i had to scrape data from multiple pages and links so for that i needed a fast crawler because the links i needed to scrape were over 6000 six thousand, yup but now that i am learning Scrapy i realized that why i was learning BeautifulSoup in the first place? i should have gone for Scrapy and i know that Selenium is for Java scripted websites and use to automate browser but i am still learning Scrapy and maybe it could also do what selenium does. So just for the sake of time saving i dont like to waste my time learning all of these libraries and instead wants to go for the most effective one. So, guys help me out.
Thanks


r/scrapinghub • • Sep 08 '20

I am Looking to buy Linkedin data (huge datasets) email [email protected]

0 Upvotes

I am Looking to buy Linkedin data (huge datasets)

email [email protected]


r/scrapinghub • • Apr 16 '21

Scrape about us section

0 Upvotes

Hey Everyone,

I'm looking for a way to scrape the "About Us" section or page of websites. Do any of you have some resources on this?

Thanks and have a good weekend!


r/scrapinghub • • Apr 12 '20

Scraping cars advertisement - looking

0 Upvotes

Hi all,

I developed a scrapping tool to catch cars advertisement on a French Website. Unfortunately this website changed their security parameters and I am not able to run my tool anymore. For sure there are other solution to catch these advertisement but in between I got a new job and have not more time to get deep into it.

I hesitated to post this on upwork or equivalent, but I imagine that due to the current situation a lot of student for example are available and looking for small/temporary/to from home jobs.

So if you are interested and have basic French knowledge please contact me.

thanks and happy easter

Tao

edit: for a better understanding, I changed some words


r/scrapinghub • • Feb 25 '21

Welcome to our new subreddit! Scrapinghub is now Zyte!

Thumbnail
self.zyte
3 Upvotes

r/scrapinghub • • Jun 04 '20

How to extract tweets from Twitter with Octoparse in 5 minutes

3 Upvotes

r/scrapinghub • • Jun 13 '20

A bot could never beat that level of obfuscation

Post image
3 Upvotes

r/scrapinghub • • Jan 15 '18

Need help with webscraper.io pagination!

0 Upvotes

Ok so i will try to describe what i want to do: Main website > list of houses with pagination > information about the house when pressed on one of the houses.

This is how i did it now. I really hope someone can help me because it stops scraping after scraping 1 page (of the list of houses).

Thanks alot in advance. Image of my settings: https://imgur.com/a/hV3iT


r/scrapinghub • • Sep 29 '17

First project, turn a blog into an ebook. Help me to overcome some obstacles.

0 Upvotes

This is my first project. I'm trying to turn some blog articles from a wordpress page into an ebook.

I downloaded the page with wget --mirror, deleted some html files that I don't need and found hat ebook-convert might be the right tool to turn the html pages into a proper epub file.

But before I do the conversion, I'd like to do some cleanup on the files, remove the navbar, comment-section and footer. I also need to convert some image src to the local folder, because wget convert-links missed them.

In order to remove certain sections in the html file I found hxremove from the html-xml-utils. As adviced in the man I ran hxnormalize -xe first for proper formatting.

Unfortunately when using hxremove it breaks the page and it doesn't get rendered in the browser anymore.

I ran hxremove footer < foo.html > bar.html

When comparing the outcome, I noticed that hxremove not only removed the footer but seemed to make changes all over the file, the formatting is different, parts get removed that I don't wanted to be removed, weird stuff. running hxnormalize afterwards didn't help either.

I suspect that the formatting of the input html file is somehow different than what hxremove is expecting and that this makes it do all this weird deletions and changes. But I have no idea how to fix this. Any ideas?


r/scrapinghub • • Feb 24 '18

Facebook Fan Page Photo + Video Exporter

0 Upvotes

Hey guys, we developed a tool that lets you export all photos and videos (in HD!) from any Facebook fan page in order to curate a massive content library.

To access the tool, all you need is a free Spektrol account. You can then use the tool here. Enjoy!


r/scrapinghub • • May 03 '17

Anybody know how to scrape data off pdf retail catalogues?

0 Upvotes

I want to be able to scrape data off pdf catalogues. An example is something like this

I assume that there is a general pattern to this, but I have no clue on how to approach this


r/scrapinghub • • Feb 13 '18

request: pls help identify a couple of CSS selectors

0 Upvotes

I have a feed, here: https://twitrss.me/twitter_user_to_rss/?user=Tom_S_Ashton/lists/outdoor

This contains a series of tweets from Twitter.

I'd like to be able to scrape the key information contained in the tweets: the user name, the content of the tweet, and ideally, a link (to the tweet).

I think I just need the CSS selector and attribute (optional) for these.

If anyone can help, it's much appreciated.

Tom


r/scrapinghub • • Sep 17 '17

When code breaks, sometimes it's NOT your fault (rant/self-dope-slap)

0 Upvotes

Previously I wrote a little Python scraping code to find my Fastrak Balance

It stopped working tonight. WTF?

I was trying all sorts of **** to get it to work. I was researching Javascript, Jquery, trigger, Action_Chain, etc.

I've spent 3 hours on this trying to debug this.

Guess what...

It's the Fastrak website that's screwed up, NOT MY CODE!!!!!

I assumed that since the Login Page I usually use to login is no longer there (it redirects to a "News" page) I had to login via some fancy action_chain and I spent the last 3 hours figuring out the syntax and all (I'm a neophyte at Python)

I was sure I had the whole thing perfect, but it redirects to the "News" page AGAIN.

THEN I did what I should have done when I first noticed the problem... Trying to login manually. :-P

Then I would have noticed that even with the right credentials typed in, the login kicks me to the News page. The ENTIRE Fastrak website is toast right now. Sunday morning.

SOMEONE picked an interesting TIME to mess with a production server.

And I wasted three hours debugging something that wasn't broken.

Duh-me.


r/scrapinghub • • Mar 12 '21

Sharing scraped data with your clients(customer) on Scrapy Cloud?

0 Upvotes

Hi,
I am new to cloud computing, I know the basics and I recently started using scrapy and scrapy cloud for web scraping. I was wondering if I am doing a scraping job for my client and instead of downloading the data myself, I can just give him a link where he or she can download the data i just scraped. Now I know you can make your data share on Scrapy Cloud to everyone but I want to know if their is a more private way to share data with a specific person who can access the database with a password. Or do I need to get services of another cloud host?


r/scrapinghub • • Jul 24 '18

Looking to Hire

0 Upvotes

I have a list of URLs that I need scraped, around 500 websites.

PM me for more details