r/learnpython • if MacItaly: return helpful / else: return to_Google • Nov 24 '20

Beginner/Intermediate Python Programmer Finally Using Multiple Files for Project

Holy Moley! I started playing around with building a mini-project using multiple files instead of one long file. Wait, let me back up...

I just started Python about a year ago, self-taught, and have been actively building desktop apps for my company in the CMP Semiconductor field. They're efficient applications that my boss loves and have been able to make us some good $$$. But, my codes are over 10,000 lines long. And I knew there had to be a better way.

Fast forward to today where I built a main file, and put a class in another file with some basic math functions. Then in my main file I built a little GUI to solve simple math problems. (I LOVE PYSIMPLEGUI, a big shout out to u/MikeTheWatchGuy for all the work he does.) And I just imported the class into my main and called it as normal...and it works. Easy as pie. I couldn't believe it. I feel like I've just learned a super valuable piece of information that I had been missing out on. Milestone achieved.

Thanks for reading and sharing in my epiphany!

433 Upvotes

75 comments sorted by

61

u/valkyndev Nov 24 '20

Hey! Curious to know, what exactly are are you writing that's 10,000 lines long? I've been trying in my own time to start work on a project that can reach that can kind of scale, but I don't know where to start

68

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 24 '20 edited Nov 25 '20

It's a semiconductor wafer endpoint detector.

These tools polish wafers and it's important not to overpolish. So based on the different types of the wafer, wafer size, whether we're measuring types of endpoint algorithms of the tool, it reads data that converts and displays realtime graphs. The program watches, calls endpoint, and signals to the tool to stop polishing. (This is a VERY general rundown of the program)

I also have built-in tools for post-processing of wafer data, exporting data, and a whole host of additional features.

5

u/[deleted] Nov 24 '20

have you considered using labview?

5

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 24 '20

I use A LOT of matplotlib, numpy, and scipy.

I don't know much about labview, or have any experience with it.

2

u/selah-uddin Nov 25 '20

*sad panda noises*

4

u/kingsillypants Nov 25 '20

in high voltage physics class, that's what we used.

Also, I'm not allowed in the lab alone. Which is fair. :P

18

u/valkyndev Nov 24 '20

Very interesting, Python seems very useful for working with large sets of data. I've just started learning myself and there seems to be an abundance of libraries to easily pull from

18

u/[deleted] Nov 24 '20

[removed] — view removed comment

5

u/valkyndev Nov 24 '20

Sorry if this is too simple a question, but out of curiosity, for your particular use, how do Python's features/libraries help? What're your exact analysis needs?

17

u/[deleted] Nov 25 '20 edited Nov 25 '20

[removed] — view removed comment

6

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

Nice. Clear and concise!

1

u/HardstyleJaw5 Nov 25 '20

I do molecular Dynamics simulations of channels so I routinely work with patch clamp data, glad to see python helping on your end too!

1

u/CarbonTubez Nov 25 '20

How are you retrieving the real-time tool process data into Python? I work in the field but use offline SQL data.

2

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

Proprietary hardware configurations.

2

u/sxh967 Nov 25 '20

Imagine if your company was in Poland and you had a trademark dispute with a Polish ice cream manufacturer.

-11

u/[deleted] Nov 24 '20

Now explain why your employer puts hundreds of thousands of dollars of machinery and product into the hands of a Python noob?

15

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 24 '20

Because my results produce $$$, and that's all anyone cares about. They don't care how I get there, as long as I get there.

Edit: As for me, I want to continue to learn and grow in my Python skills. So this milestone I've overcome, I feel is great for my personal growth and self-efficacy in my programming skills.

6

u/[deleted] Nov 24 '20

Can’t argue with results! Keep it up. I imagine programming is a natural branch to your trained discipline? Industrial process engineer?

10

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 24 '20

I have a dual Computer Engineering/Computer Science degree from 2002, then law school (I hated law), then a Masters in Education. I left tech to teach middle school math for 10 years. But I couldn't afford my family because we pay teachers like garbage, so I came back to tech 2 years ago.

My title is Software Engineer and I use Python, ladder logic, all sorts of proprietary PLC languages, HMI code, and a few others. It's been a great journey and I really enjoy it.

Edit: I loved teaching, and this is the type of teacher I was:

Action Fraction math video

And yes, I am Action Fraction.

2

u/kingsillypants Nov 25 '20

congrats man!

3

u/Hmolds Nov 25 '20

I can understand from a new beginner's perspective that the number of lines is impressive. However, the line count doesn't solve your problem. The code does. Having much unnecessary code leads just to problems down the line.

Check out the "Stop writing classes" talk by Jack Diederich where this genius quote came from.

I hate code, and I want as little of it as possible in our product.

41

u/shuttup_meg Nov 24 '20

Finally Using Multiple Files for Project

You are stepping up to "modules" and "packages." Check this out:

https://realpython.com/python-modules-packages/

15

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 24 '20

+1 for you!

Fantastic! This will provide clarity and a concise guide for further progression. Thank you!

4

u/[deleted] Nov 25 '20

Also, learning a language like C# has really helped me learn about encapsulation, inheritance, polymorphism and abstraction. Even though Python technically is "OOP", a language like C# where it's strictly enforced and has different class files, etc has really helped me with structuring and breaking up my files.

1

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

In my free time I've been learning C#. Trying to take my programming to the next level.

2

u/rocket_kielbasa Nov 25 '20

Hey man 10000 lines of code that is AMAZING! I’m also trying to create a data dashboard sort of thing, could you please tell me how/what you used to make the real time graphs? And can I display the graph using the module you mentioned?

1

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

I use matplotlib. I use the interactive mode and collect the data and display it in real-time.

Here is a mini-tutorial of creating an interactive plot in matplotlib. I hope it helps...

https://matplotlib.org/3.3.0/tutorials/introductory/usage.html#sphx-glr-tutorials-introductory-usage-py

1

u/barbequed-code Nov 25 '20

One more thing, that, for some reason neither that excellent article nor PyPi docs mention, once you put your package's source on git, you'll want to install from that repo:

pip install -e /path/to/local/clone/of/repo

Then, whenever you update the code in your repo, your subsequent imports will automatically use that code.

5

u/Finally_Adult Nov 24 '20

Will pyinstaller (forgive my terminology here, just started playing with python apps yesterday) compile all the modules you create automatically?

3

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 24 '20

pyinstaller -F yourscript.py

The -F tells PyInstaller to compile it into a single .exe file. So all of the additional python files that I import into the main file are pulled into a single executable file.

5

u/shuttup_meg Nov 24 '20

Yep, pyinstaller, py2exe, etc know how to find all the modules and packages you use and bundle them with your main.py app. Sometimes you have to give them hints in their config/setup, but there are examples.

3

u/[deleted] Nov 25 '20

Keep in mind there was an issue with pyinstaller and python 3.8, unsure if it has been fixed but I have switched back to 3.7 python and have no issues.

1

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

YES! It's with pyinstaller. I went through a number of pyinstaller member forums and found out how they are working around it. If you'd like to know just send me a message and I'll walk you through it.

9

u/SadFrodo401 Nov 24 '20

A path for a beginner on python to follow if you can please share.

11

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 24 '20

I made this post a while ago, it should definitely lead you in the right direction...

https://www.reddit.com/r/ToeKneeKnows/comments/imjq2u/to_anyone_looking_to_learn_python_python/

3

u/sxh967 Nov 25 '20

Codecombat is effing awesome. If/when I have kids I hope I can get them onto this.

1

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

Yeah, that one is a lot of fun. I'm glad you're enjoying it!

2

u/SadFrodo401 Nov 25 '20

Thank you so much for sharing!

8

u/austinmakesjazzmusic Nov 25 '20

Don’t you mean easy as Py?

Congrats on the milestone!

3

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

Brilliant word play!

6

u/mathmanmathman Nov 25 '20

But, my codes are over 10,000 lines long

Holy JESUS! Obviously, you've already realized this is a pain, but please, for the sake of anyone that might maintain your code in the future, split that into way more than two files!

6

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

I’m the only software engineer at my company using Python. So it’s all me maintaining it. So I know how much of a pain it is.

2

u/Hacker_wana_be Nov 25 '20

Wait what. I always wanted to do that for my new project but i get laZy to try it out

2

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

nO nEeD To bE laZy, eXpiReMEnT and LeaRN!

2

u/Prof_P30 Nov 25 '20

I know using globals is a bad thing to do, but how to avoid it? I want to display program name and version in different scripts of my program. But defining only once. How do you do that?

3

u/MikeTheWatchGuy Nov 25 '20

Make a function that returns those values. When a part of you programs wants the info, call the function.

4

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

Here he is! I am still humbled by your continued generosity of knowledge and willingness to share and help others grow within their coding journey. You've helped me on Reddit, Github, and via email. I thank you again for your assistance, and for your tireless work in developing PySimpleGUI.

1

u/MikeTheWatchGuy Nov 25 '20

☺I'm just a developer guy like many others here. Lots of people want to help.

It's nice that you're grateful for the help. Gratitude and happiness are linked, so I hope you're as happy as you sound grateful. I appreciate the recognition, but it's not needed. Seeing what other people create using my stuff is enough recognition.

2

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

I wish I could give you a tutorial on how I've used your stuff to build my programs. It's been a huge benefit to my company, and to my personal growth as a developer. And there's something about building a quality program that boosts self-esteem and self-efficacy. I am indeed happy and grateful.

1

u/Prof_P30 Nov 25 '20

Thanks.

And what to do with thousands of these, which I user over all my scripts/modules in dialogs, main windows, widgets across my projects.

const.py:

...

ICON_ADD = "./img/icons8-add-new-50.png"ICON_EDIT = "./img/icons8-edit-file-50.png"ICON_DEL = "./img/icons8-cancel-50.png"

...

Making a function for every single entry seems a bit of an overdose?

1

u/MikeTheWatchGuy Nov 25 '20

One way could be to use a dictionary that's filled with these values. I've seen some programs use a class to hold constants.

1

u/MikeTheWatchGuy Nov 25 '20

This is what I meant when I said using a class to hold constants. The class doesn't have to be instantiated in order to get the values. This way it's not actually a variable taking up space as a global.

class consts():
    SIZE = (5, 3)
    BUTTON_NAME = 'Go'

print(f'the size is {consts.SIZE}, button name is {consts.BUTTON_NAME}')

6

u/AlexandrTheGreat Nov 24 '20

In my quest to find a WYSIWYG GUI editor for python, I found pysimplegui which is lovely, and while it doesn't fit the original need, it does make prototyping very easy.

1

u/buttwarmers Nov 24 '20

Have you tried qt designer?

1

u/AlexandrTheGreat Nov 24 '20

I did, but it spat out weird code and IIRC it required a few hoops to jump through to use it beyond a mock-up. IIRC it also could really be updated? I tried it out a long-ish time ago

2

u/Dogeek Expert - 3.9.1 Nov 25 '20

The workflow with Qt Designer is to save the .ui file somewhere, and then use the uic command line utility to "compile" that into a python file.

That file is not intended to be used as is though, as a recompile would overwrite it. You can import it though, and make a new class that inherits from it with all the methods and signals you want/need. That way,, if you need to update the UI and recompile it, you don't lose any code.

1

u/AlexandrTheGreat Nov 25 '20

Hmmm, thanks. I'll give it another try

1

u/MikeTheWatchGuy Nov 25 '20

This seems like the right way to go about using the Qt designer (or any designer), but I don't recall seeing any tutorials that follow this approach.

1

u/[deleted] Nov 25 '20

Self-taught, huh? What resources did you use?

3

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

1

u/garlic_bread_thief Nov 25 '20

Do you by any chance write softwares where you make interactive GUIs like not just having text boxes, fields, labels, but also other objects? I'll be trying something advanced like that soon but not sure what module to use.

3

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

So my program utilizes multiple types of windows, and PySimpleGUI offers flexibility in how your GUI looks. What types of objects are you looking to use?

1

u/garlic_bread_thief Nov 25 '20

Like graphs. The user should be able to click on the graph and my program should read from the graph to do further calculations.

2

u/MacItaly if MacItaly: return helpful / else: return to_Google Nov 25 '20

I use matplotlib to generate graphs. If it's a static plot, matplotlib allows you to trace lines for (x, y) values. You could also embed the matplotlib plot into a PySimpleGUI window with buttons along the side that, when clicked, would do whatever calculations you're referring to.

1

u/MikeTheWatchGuy Nov 25 '20

You can get click locations within a Graph Element. You'll need to do your own processing to determine what that click meant. To help you determine what to do, you can get a list of items previously drawn that are under the point you clicked by calling Graph.get_figures_at_location((x,y)).

1

u/Volocinator Feb 28 '21

To clarify, you made the desktop apps entirely in PySimpleGUI? I was wondering if you made the desktop apps executables, thanks!

1

u/MacItaly if MacItaly: return helpful / else: return to_Google Feb 28 '21

PSG is the GUI I used. But I use lots of other modules as well, then I compiled it all into an executable.

1

u/Prof_P30 Nov 25 '20

Could you give an example on how you organize and access the different files over different folders?

I am using global constants in a file called const.py; here I am defining e.g. icon file names, the application name, the application version an so on to be reused on all my GUI scripts.

And now the problem begins...

I wanted to make subfolders for my dialog Ui files, for my main windows, for my views...

Now I don't know how to access the file const.py from a hierarchical tree structure, sometimes need to jump two dirs out, sometimes one, etc.

import .const vs from .. import const etc.

1

u/Wakusei69400 Nov 25 '20

I'm amazed that you managed to work with10,000 lines long files. What didn't you split your code before ?

4

u/firedrow firedrow Nov 25 '20

As another self-taught Python guy, understanding how to write your own modules/libraries can seem intimidating, especially when you are figuring it out yourself.

Then one day you come across a Reddit post with a simple example or explanation, and a little switch clicks and you go find your biggest file and break it down.