r/learnpython • • Feb 03 '21

My eyes have been opened!

Only 2 weeks into learning Python and I’m already dreaming of what I can be automating. Loads of help and ideas in here, so thanks to everyone that contributes!

I’m enjoying the challenge of working things out, but my 1st plan is to read a pdf, extract certain bits of data and transpose into a predefined template.

Already cracked pdf reading (thanks to the course by Al Sweigart!), so onto the next step.

Any thoughts or tips on how or where to store data once I’ve extracted text from a pdf would be appreciated!

Edit: thanks all for the comments support and ideas. Really appreciated!

A few people have asked what resources I’m learning from - I’m using the youtube beginners guide by Mosh and and Automate the Boring Stuff by Al Sweigart on Udemy. They cover similar topics but in different ways, so I’ve learnt from both. Not to mention this group, I’ve been a lurker for a while and this is my 1st post.

I do have the benefit of working in a data driven industry, so I have something practical and tangible to work towards. People have different learning styles, but I think I struggle to learn without having a project that “means something” to me.

So I guess that would be my takeaway - don’t just expect to learn python by taking a few courses. Learn the basics, but try to apply some practical examples as soon as you get your head around the logic!

464 Upvotes

48 comments sorted by

20

u/JozsefPeitli Feb 03 '21

You mean permanently in a file or temporally. I think the easiest way fkr file is csv, for me temporally is pandas dataframe.

7

u/th3_north3rn_monk3y Feb 03 '21

Great stuff, thanks. Will take a look at Pandas, once its in a template it can be deleted I think.

5

u/JozsefPeitli Feb 03 '21

Of course. Del df

5

u/staged84 Feb 03 '21

Hi, I'm trying to teach myself python but kind of lost. Been watching YouTube videos from mosh lately. Can you please let me know what worked for you and what didn't?

Thanks!

17

u/th3_north3rn_monk3y Feb 03 '21

Yeah of course!

Whats worked for me is having a plan to do “something”!

I did the 1st few sessions with Mosh to get the basics, and found the theory similar to excel formula.

So I came up with a project. I learn better with practical examples that mean something to me, the Mosh tutorials are great but the one thing I figured pretty quickly is that, similar to excel, all you need to really do is ask your search engine a question.... theres so many people out there with similar projects or ideas.

So in my day job, I analyse data, but our biggest challenge is getting the data “in”. Knowing we get a lot of data from pdfs and knowing I can use Python to scrape data, my mind started wondering and now I can scrape data from a pdf.

I created a flow chart to help keep me on track, so have a few other bits to learn:

  • How do I log into a portal and download a pdf

  • How do I transpose the date scraped from a pdf into a predefined format

  • How do I save the pdfs for future reference

Etc etc.

Hopefully, one day, I’ll have an answer to most of these and will be able to combine the code into one!

Thats my plan anyway!

8

u/garlic_bread_thief Feb 03 '21

How do I log into a portal and download a pdf

You can try selenium to automate the browser. Works like a charm.

7

u/th3_north3rn_monk3y Feb 03 '21

Ahaa! Thanks, I’m literally reading up on Selenium now, so good to see I’m on the right track.

12

u/izaakstern Feb 03 '21

Pro way would be the requests lib...a lot faster without a browser

5

u/Cuetzalcoatl Feb 03 '21

I have been getting so much help from the python community that I literally sent a message the other day to a redditor that basically took my code and rewrote it, full with annotations and comments. It was heart melting hahaha

I have been getting this notion, and would love your opinion: selenium for me is for dynamic pages that are finicky and complex, so you actually need a visual client to help guide your code; and requests is usually for more static pages or pages that you can easily get beautifulsoup to parse. What do you think?

4

u/izaakstern Feb 03 '21

Yeah, it depends on the webpage, and what you wanna do. But its mostly nothing for a python beginner. I do a lot of automation for years and it does not matter for me, if the page is complex or not. I always do, what I want to do on the page with a sniffer running in the background (I am using Fiddler). Than I just have to rewrite the client -> server calls (with a lot experience of headers, cookies and stuff)

6

u/Farmher315 Feb 03 '21

Hey! I was a total beginner and I just jumped right into Automate the Boring Stuff with Python, it's a book and he has a Udemy video lecture to go along with it. Not having any background in much computer tech stuff, I found his method of teach very helpful. Many times, I'd have to pause and go research something I did not know about but I think that helped my learning process alot.

2

u/Encrypt-Keeper Feb 04 '21

What I did was come up with a project first, then learn enough to implement the one project. Start thinking of new features to add then learn how to do those. Small chunks, don't choose "Make a gui application" at this stage. Eventually you'll feel out where you lack in a fundamental, with thoughts like "What should be a function or just code". "Should this all go in this file or should a make a module". "How do I import my module?"

Basically, treat it like a jigsaw puzzle. Just start with a random piece and find a piece to fit it in. You don't have to know all the pieces or what the final picture should even be. But the more "single things" you can do in python, the clearer the picture of it all is, and you become more aware of what you're missing, and now your good at finding specific pieces instead of random ones and it becomes easier and easier to add more to your repertoire.

2

u/[deleted] Feb 04 '21

Python for Everybody and anything else by Chuck Severance, it's on youtube.

3

u/cagtbd Feb 03 '21

I want to learn python too, are you learning by yourself or following a course in order to do your projects?

4

u/th3_north3rn_monk3y Feb 03 '21

I’m just learning as I go really, but have completed the youtube tutorial by Mosh and currently going through the Al Sweigart course.

I’ve worked with data and systems as an analyst most my working life, but its mainly been the outputs.

However, we’re only a small team, so its all hands when it comes to data in.

So, Python is basically answer to a question - how do I make this easier. Having an idea in mind of what I want to achieve has really helped.

3

u/cagtbd Feb 03 '21

Thank you so much, I'll give it a try, this year I need to learn python and R in order to be more proficient in BI.

1

u/th3_north3rn_monk3y Feb 03 '21

Definitely suggest having a working example that means something to you, but the basics in the course do really help. I didnt know what the “print” function did until 2 weeks ago, but once you figure out the logic it really starts to make sense.

2

u/cagtbd Feb 03 '21

Thanks, there is a python free course provided by sololearn, you can download the app and when you finish you can get a certificate. Check it out, it helped me land my former job even when I never used the languages they asked.

3

u/TheUruz Feb 03 '21

congrats buddy, i had to learn the pdf part all by myself and it sucked. pdf is one hell of a format... anyway if what you are extracting is just plain text (so you don't need to save it as a schema of any kind) just drop it in a simple text file, if you want something more classy because your data has a logic behind save them in a .json file or a .yaml file (wich are pretty handy to parse and use in your code) if you have tons of data to store and you want them in the most safe place on earth, as someone is suggesting, upload your data to a bucket in the cloud (AWS or GCP it doesn't really matter for a small project like this)

may i ask wich library are you using to parse pdf?

1

u/th3_north3rn_monk3y Feb 03 '21

Cheers!

Seems I have loads of different options, I’m thinking I’ll export into csv for now, while I figure out how to transpose into my excel based import template... then maybe figure out how I automate that part... baby steps!

I used PyPDF2 to parse the pdf. I’ve started on a very basic pdf so far, plan to test on multiple variants once I get an easy one working.

2

u/TheUruz Feb 03 '21

got it. i've tried that too but it wasn't the best to parse non machine generated pdf so i swapped to PyMuPDF wich i'm pretty satisfied with. if you'll ever have trouble with PyPDF2 on more complex pdf give it a try ;)

1

u/th3_north3rn_monk3y Feb 03 '21

Thanks for the heads up! Will add it to a resource list I’m pulling together. The pdfs I’m testing with are very basic, so expecting a bit more of a challenge when I get into more complex data.

3

u/The_nemea Feb 03 '21

its pretty legit to learn it. I have been holding off on a lot of projects over the years because I couldn't code. started learning python and a week and a half later I wrote a program that would take a list of company names, search google, determine if there was a website associated with the name, pull the website address and phone number, and throw the positive results into a spreadsheet.

took a couple week job with thousands of companies and did it within a couple hours.

I think a .csv or an excel spreadsheet would probably by your best bet for storing info depending on what it is.

1

u/th3_north3rn_monk3y Feb 03 '21

Cheers, its really got my head thinking anyway. All those small tasks that take 5 mins here and there really add up. My list of “things to try” is growing anyway!

I think csv is the way forward for now, as I have a template in excel I need to get the data to work with... will keep thinking anyway!

78

u/Mobile_Busy Feb 03 '21

You can write them to flat files with file.write(), use pandas to create an excel spreadsheet, or learn to use SQL and store it in a proper database.

21

u/garlic_bread_thief Feb 03 '21

Are flat files like simple text files (.txt)?

30

u/Mobile_Busy Feb 03 '21

They can be. Most of the time when I use the phrase, I am referring to comma-separated value files (.csv).

7

u/20_ADG_02 Feb 03 '21

I've never looked it up, thanks for the input

5

u/[deleted] Feb 03 '21

As far as I know, and please correct me if I'm wrong, but files don't necessarily need an extension associated.

My understanding is that an extension is basically a pointer to what program can interpret the file. In some form or another, all files are text files.

If you've dipped deep enough into file systems for different programs, you certainly came across generic files, which you can open with a notepad. Their encoding might not always be interpretable by the notepad though

3

u/max123246 Feb 03 '21 edited Jan 30 '26

This post was mass deleted and anonymized with Redact

bright roof wise literate consist obtainable distinct skirt chop sink

2

u/Mobile_Busy Feb 03 '21

In Windows they kind of do. In *nix it's more of a convention.

https://en.m.wikipedia.org/wiki/Filename_extension

6

u/Peanutbutter_Warrior Feb 03 '21

Presumably so. Flat doesn't really mean anything in relation to files, at least afaik

13

u/Mobile_Busy Feb 03 '21

I love learning new things. Don't you?

https://en.m.wikipedia.org/wiki/Flat-file_database

3

u/VBoB13 Feb 03 '21

Oh! So kinda like SQLite then? 😅

3

u/Mobile_Busy Feb 03 '21

No. SQLite supports indexing.

2

u/neriad200 Feb 03 '21

what a fancy way to describe it

3

u/raias4 Feb 04 '21

What would be some advantages or disadvantages of using an excel spreadsheet versus SQL?

0

u/Mobile_Busy Feb 04 '21

If you're just starting to learn, do them both. In the real world, some of the considerations would be the size of the dataset, who will be using the data, what they will be using it for, and how they will be using it.

8

u/djavaman Feb 03 '21

AWS automation with python scripts is pretty cool. And easy.

So, maybe upload the text to an s3 bucket.

3

u/robot_ankles Feb 03 '21

/u/th3_north3rn_monk3y, his eyes opened!

(Off topic, but there has to be significant Venn Diagram overlap between this sub and ST:TNG viewers.)

2

u/HasBeendead Feb 03 '21

lol, more like reborn

3

u/AlbatrossFun8542 Feb 03 '21

I would say writing to a txt is fairly simple in python and is a decent way to store data if it is extracted from a pdf.

2

u/whatisitcousin Feb 04 '21

What are you learning from?

2

u/cab0addict Feb 04 '21

Depending on what you want to do with it, you could create a Json (or insert structured format here) file and store them in mongo or some other document store.

1

u/AmatureProgrammer Feb 03 '21

Nice, what curriculumn are you using to learn? is it the course by Al Sweigart that you mentioned? If so, is it a udemy/youtube channel?

1

u/GodIsInsideOfYou Feb 04 '21

Check out opencv