r/java • • Mar 31 '22

Introduction to Web Scraping With Java

https://www.scrapingbee.com/blog/introduction-to-web-scraping-with-java/
327 Upvotes

38 comments sorted by

4

u/stregnozis Apr 01 '22

I wrote parser on java. And it works well and main advantage of it is easy to expand its functions. Yes, I can say that in python I can do the main functions more quickly, but when I need some extras - java is great. I love java

1

u/ThisPlatformIsBad Apr 05 '22

What's the pros/cons between this and using JSoup?

0

u/[deleted] Mar 31 '22

[deleted]

35

u/[deleted] Mar 31 '22

I’m writing a web scraper in Java right now. Using MS Playwright. I’m old enough to remember when the script kiddies used Perl and they made the same kind of noise. “Oooh. Java. Gross”. I wonder where they are today. Ah well. Back to the grind…

10

u/lechatsportif Mar 31 '22

They grudgingly admitted no one wanted RoR apps anymore after they pivoted to that, leaving behind messes Java consultants get paid top dollar to port to Spring. These days however they're in Go because... least its not java!

2

u/NewSchoolBoxer May 02 '22

RoR, that takes me back. Was the HOT NEW THING circa 2010. Thought I was going to have to learn it, then it suddenly disappeared. I'd be interested in reading about its rise and fall. Heh I got paid to port Java 5 old Spring + SOAP to Java 11 Spring 4 + REST.

The rise of Go outside Google surprised me. I talked to Microsoft devs who use it as the required language on some projects. Other projects let you roll with most any OOP.

0

u/[deleted] Mar 31 '22

[deleted]

1

u/unknowinm Apr 20 '22

at least you not waste time trying to align stuff with tab or waste time debugging runtime errors...go just feels java 5 to say the least and you don't have proper error handling and pointers???

1

u/NewSchoolBoxer May 02 '22

10 years in Java on the job here. Eclipse workspace induced compiler errors are hell. IntelliJ better? Tell that to corporate and then force people to switch.

Language is verbose which is amazing as a beginner but not in dealing with 20k LOC repositories. Big disadvantage in "language agnostic" timed coding tests. Early bad API decisions haunt me when I run into Hashtable or old Date manipulation with longs. Generics offer no performance gain. A hard break around Java 5 would have nice.

Sure, first coding language, Java is fine and has many (the most?) job opportunities. I'd rather do Python but the job market is more niche. Too bad it's usually stuck to slow-ish interpreter. I'd rather do C# with best in class Visual Studio, pseudo structs and no get/set bloat.

I shudder at JS too. Sometimes Python can be too efficient for its own good when it comes to understanding what's going on inside. Never used Go.

But yeah, some Java hate because the biggest players are always carrying a target on their backs. People hate on C# because of Microsoft. I hate on IBM myself...

10

u/Worth_Trust_3825 Mar 31 '22

Funny how boring tools survive.

4

u/buzzsawddog Mar 31 '22

They wrote a bunch of code that I now have to maintain... Not a happy camper...

1

u/neopointer Mar 31 '22

I think you didn't mean it, but just wanted to understand. Are you using playwright with java? If yes, how?

3

u/burly_griffin Apr 01 '22

Not OP but also not sure what you mean by "how". It has first party support and the build plugin sets everything up. It's also literally the first result when searching for it.

https://playwright.dev/java/docs/intro

28

u/JustWaveNSmile Mar 31 '22

That will be me, im working on a Spring boot react project that scrap games from ps store to notify when my favorite games are on sales with the price I choose

6

u/cosmodisc Mar 31 '22

If you can do stuff like this,I somehow doubt you need a discount for your favourite games:)

26

u/Regis_DeVallis Mar 31 '22

Just because we can do stuff like that doesn't mean we're not broke.

4

u/blakeman8192 Apr 01 '22 edited Jun 26 '23

.

5

u/Regis_DeVallis Apr 01 '22

That's true, but if only it was that easy to get hired.

5

u/blakeman8192 Apr 01 '22 edited Jun 26 '23

.

7

u/Regis_DeVallis Apr 01 '22

I mean I'm a full time student right now, so I haven't tried getting a real dev job, as I can pretty much only work summers. But hey I'm spending my free time hammering away on rails projects on my GitHub so when I am out of school I have a nice portfolio to show off.

5

u/blakeman8192 Apr 01 '22 edited Jun 26 '23

.

4

u/[deleted] Mar 31 '22

I have a vault of articles saved as screenshots by using playwright behind a simple api. For when I want The whole visual not only html, and nowadays many websites load content via java script so I need a “browser”. Everything is on the web nowadays so even more reasons for having options to save and process it I argue :)

Will definitely read the article!

10

u/Worth_Trust_3825 Mar 31 '22

Who wouldn't? What do you propose using instead? Does it matter if you interface with selenium with firefox in ruby or c#? Is there a quality difference in using prolog over cobol? All you do is write to a socket, and read from a socket, then process, and write to another socket. For all you care, you can do all of that manually. Your goal is not to use java, after all.

5

u/GrayDonkey Mar 31 '22

Puppeteer under NodeJS is the industry norm when you need to support JS rendering.

HtmlUnit is an interesting choice, usually I use JSoup for Java based scraping. It's faster than Puppeteer but you have to ensure the content is static or uses server-side rendering.

3

u/Worth_Trust_3825 Apr 01 '22

Again, what's the difference between chromedriver/equivalent to puppeteer? You're running an entire browser under the hood anyways.

2

u/koreth Mar 31 '22

At a previous job I had to build a bunch of web scrapers and Puppeteer was far and away the best tool for the job. I tried JSoup and HTMLUnit and they are somewhat acceptable for scraping simple static content, but good luck getting those to, say, navigate a React or Vaadin site.

-10

u/Nymeriea Mar 31 '22

With all the Js rendering, I guess electron will be a better choice

5

u/berry120 Mar 31 '22

Electron is never the better choice :-)

2

u/ceph12 Apr 01 '22

This is a marketing material for a saas product

1

u/_litecoin_ Apr 01 '22

Me. With Playwright

1

u/[deleted] Apr 01 '22

I literally am lol

https://github.com/mattwelke/packt-book-bot

It was a holiday break 2021 project though.

1

u/xxkmatiasxx Apr 01 '22

What would you use?

1

u/wildjokers Apr 01 '22

I have web scrapped with java before. The honest question is why wouldn't someone do web scraping with Java?

25

u/[deleted] Mar 31 '22

[removed] — view removed comment

6

u/Tostino Mar 31 '22

Yup, I did the same years back when I needed to crawl through a few thousand pages.

3

u/humoroushaxor Mar 31 '22

I think the honest answer is dynamic typing and easier text manipulation. In the examples they need to constantly cast classes and object map json.

I'm primarily a Java dev but I find this stuff so much easier in Python.

3

u/Mats56 Apr 01 '22

It's such an annoyance speeding it up in python, though, if you want to crawl lots of pages. In java, just add new pages to a thread safe queue when you scrape some url. Have lots of workers in a threadpool picking from that queue.

In python it's a project in itself having multiple things working in parallel.

3

u/yawkat Apr 02 '22

They use htmlunit in the article for some reason, not jsoup. Jsoup requires basically no casts.

I've done a lot of scraping and I've yet to find an api that is more convenient than jsoup for it, in any language

-1

u/humoroushaxor Mar 31 '22

I think the honest answer is dynamic typing and easier text manipulation. In the examples they need to constantly cast classes and object map json.

I'm primarily a Java dev but I find this stuff so much easier in Python.