r/learnpython • • Jul 11 '21

If you are employed as a python programmer, tell me the stuff you do on a day in the office

I want to be employed not python in particular but its my favorite language so I wanted to know what level of knowledge a job requires like what the people in offices actually do? I've seen videos but its just them typing on a computer

468 Upvotes

121 comments sorted by

View all comments

234

u/epicmindwarp Jul 11 '21

I do a lot of system and data connectivity; how to move data between systems, getting different systems to speak to each other.

A large chunk of it is automating human tasks where there is no programmatic way of doing it from source e.g. grabbing CSV files from location X and loading it into Y, all pre-configured, all canned.

The value add is being able to run checks, validations, and report back on the data when it's done. That way, humans only need to look at a green status, or focus on exceptions. The boring stuff, the system deals with.

31

u/Okotscha Jul 11 '21

Exactly what I do. Broad field of Data Eng.

DB X ----> processing ----> DBY
We work with a commodity trading system, hence it's lots of forecasts, public data, message data. The processing-unit want's to be pytest-ed and cuddled every-now-and-then.

Mostly py scripts that are run by a runner and SQLs/TimeScaleDBs.

5

u/epicmindwarp Jul 11 '21

Yep, pretty much it. It's a shame we can't connect everything DB to DB!

7

u/lps2 Jul 11 '21

I'm glad we can't, it keeps me employed!

3

u/epicmindwarp Jul 11 '21

run by a runner

Which runner do you use?

→ More replies (4)
→ More replies (4)

19

u/AALLI_aki Jul 11 '21

Great reply thank you

14

u/LongtopShortbottom Jul 11 '21

This sounds like an absolute dream job! I’m very new to Python and have a moderate database background. Python is wildly flexible but this is the way I plan on using it. Can you comment more on your background, how many years of experience you have, etc.?

6

u/epicmindwarp Jul 11 '21

Was not expecting this comment to get any traction.

I have a background in Finance, but I do a lot of work in data - I'm a Quant with a data and backend focus as opposed to analytics.

A strong understanding of systems and IT (but not necessarily CompSci) is required.

→ More replies (3)

7

u/marcellonastri Jul 11 '21

u/epicmindwarp is such a great person that I'm commenting this to let him/her know.
Ty for your kindness.

4

u/epicmindwarp Jul 11 '21 edited Jul 11 '21

Oh, you.

6

u/Magestylord Jul 11 '21

This is data engineering, I assume?

4

u/epicmindwarp Jul 11 '21

I'm a Quant, with a data specialism.

7

u/Magestylord Jul 11 '21

Hey, yo. Reading the comments on this thread, I would like to thank you for taking time to answer each and every one's queries :)

→ More replies (4)

5

u/Level1Roshan Jul 11 '21

So you're saying you automate.... the boring stuff? Nice.

→ More replies (1)

0

u/[deleted] Jul 11 '21

[deleted]

6

u/epicmindwarp Jul 11 '21

Absolutely.

Not as extensively used as you might think, but definitely useful.

3

u/takeonzach Jul 11 '21

Any advice for someone who works with python but has no web experience and would like to be able to automate the downloading of some CSV reports from a third party?

5

u/epicmindwarp Jul 11 '21

Selenium, Chromdriver, HTML.

Learn how Web pages are built, and the rest is just tinkering with terminology e.g. What's a div tag, form input fields etc, and how to find objects in source code.

3

u/call_me_mistress99 Jul 11 '21

What exactly do you mean with system? Can you give an example?

13

u/epicmindwarp Jul 11 '21

Imagine downloading a CSV from Bank X and uploading to Azure Blob Storage, to read from Azure SQL Server.

Bank X has no API, so building that data pipelines is key i.e. How to get the CSV, and how to autoload it in.

8

u/call_me_mistress99 Jul 11 '21

Thank you for your response. I haven't understood anything so please tell me if my research is correct.

A Comma Separated Values (CSV) file is a plain text file that contains a list of data. These files are often used for exchanging data between different applications. They mostly use the comma character to separate data, but sometimes use other characters, like semicolons.

E.g. Name,Email,Phone Number,Address

Bob Smith,[email protected],123-456-7890,123 Fake Street

The idea of CSV files is that you can export complex data from one application to a CSV file, and then import the data in that CSV file into another application.

Bank x is a bank that has no branches and everything is done online.

Azure Blob Storage is a service for storing large amounts of unstructured object data, such as text or binary data.

A server is a computer program/device that provides a service to another computer program and its user, also known as the client.

Azure SQL Server is Azure's Server programmed in programming language named SQL.

An interface a device or program enabling a user to communicate with a computer. E.g. a graphical user interface (GUI) is a type of user interface through which users interact with electronic devices.

When you use an API, you're using an interface that lets the program you write control or access a program somebody else wrote.

As an analogy, suppose your program is a robot and you want your robot to get into a car and drive across the city.

When you drive a car, do you care how it works inside? Do you care if it's a V6 or V8? That it's a traditional gas engine, a disel, or a hybrid? All of those things affect how the car works, but those aren't the interface. The interface of a car is the steering wheel, gearshift, and pedals - plus all of the other levers and controls to access all of the car's other functions.

So if you were hypothetically building a robot to control a car, you don't have to teach it how an internal combustion engine works. You have to teach it how to accelerate, brake, steer, and shift gears, by working those controls.

So now let's relate this back to a programming example.

Twitter has an API. That means you can write a program that accesses or controls Twitter. You don't have to know how Twitter works internally. You just have to know their API - the interface for programmers to interact with their service. In the case of Twitter, it lets you write a program to do things like access someone's most recent tweets, or log into your Twitter account and post a new Tweet.

Most of the major web services like Google, Facebook, Reddit, etc. all have APIs - you can use them to automate almost anything you would do manually.

A data pipeline is a set of actions that ingest raw data from different sources and move the data to a destination for storage and analysis.

Don't know what autoload is.

TLDR:

Imagine downloading a plain text file that contains a list of data. These files are often used for exchanging data between different applications.

We downloaded the plain text file from an online bank and want to upload it to a program for storing large amounts of unstructured object data. Something with servers.

Bank x doesn't have an interface which allows outsiders to play with code belonging to bank x. So we need to build a set of actions that ingest raw data from different sources and move the data to a destination for storage and analysis e.g. how to get csv and autoload.

8

u/epicmindwarp Jul 11 '21

we need to build a set of actions that ingest raw data from different sources and move the data to a destination for storage and analysis e.g. how to get csv and autoload.

That is an excellent TL;DR, and the rest of your comment is spot-on.

The value add someone likes me provides is

  • Experience i.e. I already know what will/won't work and what areas would benefit a company to focus on

  • Knowledge of technology - I don't know everything out there, but I would know enough to be able to go away and find something suitable for your needs

  • Intricate knowledge of IT workings - I can tell you from a cursory oversight if your goal is achievable based on my knowledge of technology and finance (in this instance)

  • Crucial: Will your human-explained workflow be replicated by a robot, and if not (most likely) how can it be achieved (using available architecture)?

It's all well and good to say "I want to price this bond", but you as a human know where to look and what to look out for - how do you teach that to a machine?

That's where I make the big £££.

→ More replies (2)
→ More replies (3)
→ More replies (1)

3

u/Anowv Jul 11 '21

This sounds like a more advanced version of what I do.

Finally getting something automated and hearing positive feedback is so f—king rewarding lol

Almost worth the crushing defeat everyday

→ More replies (3)
→ More replies (15)