r/learnpython • • Jul 11 '21

If you are employed as a python programmer, tell me the stuff you do on a day in the office

I want to be employed not python in particular but its my favorite language so I wanted to know what level of knowledge a job requires like what the people in offices actually do? I've seen videos but its just them typing on a computer

468 Upvotes

121 comments sorted by

201

u/[deleted] Jul 11 '21 edited Jul 12 '21

I don't work as a programmer but work with many programmers and have managed multiple large teams (some with over 100 members, across multiple locations) of programmers over my career.

Day jobs vary greatly depending on the level and nature of the role, the organisation, and challenges being addressed. There's in no one size fits all.

So here's some off-the-top-of-my-head thoughts. I've no doubt missed loads of stuff (and fast turn-around start-up environments will leave out many of the steps I've indicated, or do them in a very efficient, fast way).

However, for the vast majority of programmers I've experience with, actual coding is a small part of their job.

Outside of small teams, a lot of Time will be spent on:

  • lots of meeting with a wide range of stakeholders

  • regular stand ups / status checks

  • coordination

Lots of workshops to agree:

  • problem/challenge being address (sprint scope perhaps)

  • what outcomes are required

  • how outputs are tested, validated, presented, consumed, stored, shared, distributed, secured, maintained, retired

  • design principles and policies

  • coding guidelines and standards

  • code testing processes

Lots of desk time (possibly collaboratively) confirming:

  • data sources / api / access arrangements / timing / flows

  • data models

  • common tools and libraries used to address this and related challenges

  • algorithm design / selection short-listing

  • searching for existing design patterns / useful code matching the preferred algorithms

  • mini proofs of concept to build confidence particular patterns / code / approaches are likely viable

  • validation against architecture, security, data access standards and patterns

  • final solution selection

  • set up of coding environment/ecosystem to code the selected solution

Then to coding,

  • in tdd environments, development and validation of tests scripts before any other code is written

  • development of solution code

  • unit testing of code

  • completion of documentation (if needed / not already self-documenting) - if you are lucky

  • submission for inclusion in overall solution

EDIT: typos and a clarification plus ps below

PS. Code submission isn't the end of the job. Typically code will then go into various levels of testing including system integration testing and in some organisations, especially in devops environments, programmers may well get heavily involved in this including working with / setting up ci/cd pipelines and test environments.

55

u/AALLI_aki Jul 11 '21

This is perfect thanks for your time

31

u/asphias Jul 11 '21

While the list above is quite complete, i'd like to add some on the technical side. Many modern developers work in a 'devops' environment where you don't just have to deliver 'code', but you are also working on deploying the code to a production environment. This can take many shapes or forms, and depending on where you work will be completely your responsibility, partially your responsibility, or it might be your responsibility not to touch anything and leave it to the Ops guys.

Tasks can include:

  • building a pipeline to run your tests / linters.

  • working on infrastructure to deploy your code to. this can come in many ways: from architecture as code, to docker containers or kubernetes clusters, to simply logging into a production machine and deploying manually.

  • including monitoring tools / logging tools into your application.

  • performance testing.

  • deploying to a Dev/Test/Acceptance environment and running your code before signing off for production.

  • creating these Dev/test/Acceptence/Production environments yourself(e.g. i could order empty prod linux virtual machines, but any configuration to do anything with them was our own responsibility.)

  • ways of testing communication with other applications- either manually on an integrated dev environment, or automatically with integration tests of some kind.

  • going onto your production environment(often a linux machine) to debug / solve stuff in production (What? you should never be doing that manually? Yeah i wish)

There are many jobs when these parts will not be your responsibility(especially when you go into data science rather than software development, although the dividing line might be vague), but often enough they are, or you're in a team where you share the responsibility and while the Ops guy in your team might know more, he's not going to do everything for you.

8

u/i_suckatjavascript Jul 12 '21

I work as a Business Analyst collaborating with different teams of engineers from different business groups. This entire description is very spot on and accurate. I use Python (not much for my role itself) for small projects that come up once in a blue moon. Very simple Python programs I create from stealing already posted examples from Stack Overflow, then modifying it using the knowledge I know from Python tutorials to make it fit for my application at work.

2

u/ughidkguys Jul 12 '21

Business Analyst here as well, second everything you said 100%

234

u/epicmindwarp Jul 11 '21

I do a lot of system and data connectivity; how to move data between systems, getting different systems to speak to each other.

A large chunk of it is automating human tasks where there is no programmatic way of doing it from source e.g. grabbing CSV files from location X and loading it into Y, all pre-configured, all canned.

The value add is being able to run checks, validations, and report back on the data when it's done. That way, humans only need to look at a green status, or focus on exceptions. The boring stuff, the system deals with.

19

u/AALLI_aki Jul 11 '21

Great reply thank you

31

u/Okotscha Jul 11 '21

Exactly what I do. Broad field of Data Eng.

DB X ----> processing ----> DBY
We work with a commodity trading system, hence it's lots of forecasts, public data, message data. The processing-unit want's to be pytest-ed and cuddled every-now-and-then.

Mostly py scripts that are run by a runner and SQLs/TimeScaleDBs.

5

u/epicmindwarp Jul 11 '21

Yep, pretty much it. It's a shame we can't connect everything DB to DB!

7

u/lps2 Jul 11 '21

I'm glad we can't, it keeps me employed!

3

u/epicmindwarp Jul 11 '21

run by a runner

Which runner do you use?

1

u/synthphreak Jul 12 '21

What is a "runner"? Are we just talking some crontab-type periodic scheduler?

1

u/synthphreak Jul 12 '21

What is a "runner"? Are we just talking some crontab-type periodic scheduler?

2

u/epicmindwarp Jul 12 '21

Basically. We use Ctrl-M

1

u/Okotscha Jul 13 '21

crontab-type periodic scheduler

In my exp GitLab Runner ist the easiest to get by.
A Docker container in which scripts are started by a a certain input.

2

u/Afrotom Jul 11 '21

Also pretty much what I do. Pipeline data from systems into the cloud for other areas of the business to use for their dashboarding. A lot of ETL with Pandas, Luigi and GCP.

1

u/TamahaganeJidai Jul 12 '21

Do you have any guide you'd recommend for starting with that kind of coding? Im no where near needing what you guys do but im going to need to do a bit of "DB X ----> processing ----> DBY" and back for a project of mine :)

2

u/Okotscha Jul 12 '21 edited Jul 12 '21

1st Step: SQL & Py Codewars. Even if you're not going to work in SQL, the querries for different DBs are all same-same-but-different. You don't want to spend 45 mins on e.g., "getting the most frequent ids in timeframe x to y" when it matters. Do this for 4 weeks 1.5hrs a day, not more, not less.

2nd Step: Host yourself a sql db on your computer via DBeaver. (this takes 10hrs min for a beginner, if you are done quicker, host it on AWS).

3rd step: Write a script that a) downloads AMZ & Baidu a stock price data (timezones) and b) only write new data to the SQL. work on this at least 25 hrs focused, if you are done earlier, do it better.
3.1 -> do pytest for all of your functions. addiontal 5hrs.

4th Step: Write a second script that computes the moving average of those two and write them to your DB, different table. work on this at least 15 hrs focused, if you are done earlier, do it better.
4.1 -> do pytest for all of your functions. addiontal 5hrs.

5th step: Write a script that when run, *alarams* (e.g., print, e-mail, sql-write in alarm-db) you when both have a positive moving average in the last 18 timesteps. work on this at least 40 hrs focused, if you are done earlier, do it better.
5.1 -> do pytest for all of your functions. addiontal 5hrs.

6th: Welcome to Docker. Have a Docker Container run above scripts three times a day. This takes forever. Afterwords, get a job.

you can send me the link to your repo after you are done with step 5, I will rev.

1

u/synthphreak Jul 12 '21 edited Jul 12 '21

Wow, this is awesomoe!

Edit: And also awesome!

14

u/LongtopShortbottom Jul 11 '21

This sounds like an absolute dream job! I’m very new to Python and have a moderate database background. Python is wildly flexible but this is the way I plan on using it. Can you comment more on your background, how many years of experience you have, etc.?

6

u/epicmindwarp Jul 11 '21

Was not expecting this comment to get any traction.

I have a background in Finance, but I do a lot of work in data - I'm a Quant with a data and backend focus as opposed to analytics.

A strong understanding of systems and IT (but not necessarily CompSci) is required.

1

u/synthphreak Jul 12 '21

systems and IT (but not necessarily CompSci)

So I never studied anything quantitative/IT/computational ini school, yet have found myself working as an ML Engineer. Life is strange...

Like you, my skills are purely amateur/self-taught. I know how to do my job, but there are a lot of gaps in my knowledge overall. So can you explain the distinction you're drawing between IT and compsci?

Compsci just seems like this impossibly broad thing to the point that referring to it all collectively is nearly meaningless. But it seems you have a different opinion. So I'd love to hear what you consider to be compsci as it relates to other disciplines where the computer is king.

0

u/[deleted] Jul 12 '21

[deleted]

3

u/epicmindwarp Jul 12 '21

I still do analytics, but my focus is more gathering the data to do the analytics, as opposed to a traditional Quant who does mainly analytics.

I.e. I build the end to end, spend 80% of my time doing data flows, and the rest, actually doing Quant analytics.

7

u/marcellonastri Jul 11 '21

u/epicmindwarp is such a great person that I'm commenting this to let him/her know.
Ty for your kindness.

4

u/epicmindwarp Jul 11 '21 edited Jul 11 '21

Oh, you.

6

u/Magestylord Jul 11 '21

This is data engineering, I assume?

4

u/epicmindwarp Jul 11 '21

I'm a Quant, with a data specialism.

7

u/Magestylord Jul 11 '21

Hey, yo. Reading the comments on this thread, I would like to thank you for taking time to answer each and every one's queries :)

5

u/epicmindwarp Jul 11 '21

Happy to help!

3

u/Magestylord Jul 11 '21

One more question, did you take a CS degree? I'm on my 3rd year of CSE undergrad

13

u/epicmindwarp Jul 11 '21

No CS degree - but I thoroughly recommend one.

I may go back and do a short CS degree just for the paper, a lot of what I do requires CS knowledge, but I have no formal training or qualifications, which makes it difficult to get through some doors that require a CS qualification to make it past the automated sections.

Having said that, I've not done bad at all for someone with no CS degree, with just my skillset gained through a keen interest and amateur experience to go on. Much of what I do, I had no prior experience of, but I knew enough of similar things for a set of good managers to take a chance on me - and it's paid off every time.

3

u/Magestylord Jul 11 '21

Damn! That's amazing. How did you join fintech companies then? Aren't they a bit demanding for degrees?

7

u/epicmindwarp Jul 11 '21

I have two degrees in finance, and a finance qualification.

I just happened to be really good at CS related stuff too.

2

u/[deleted] Jul 11 '21

doesn't sound like it (I'm currently in training)

2

u/AchillesDev Jul 11 '21

This reads similar to my experience as a data engineer for the past 6ish years that I’ve had the title.

1

u/[deleted] Jul 11 '21

cool. I'm less than a noob so you can ignore me

1

u/tobiasvl tobiasvl Jul 12 '21

My job is essentially the same thing and we call it data integration.

5

u/Level1Roshan Jul 11 '21

So you're saying you automate.... the boring stuff? Nice.

1

u/epicmindwarp Jul 11 '21

I took the book to heart. I ended up paying for it.

3

u/takeonzach Jul 11 '21

Any advice for someone who works with python but has no web experience and would like to be able to automate the downloading of some CSV reports from a third party?

5

u/epicmindwarp Jul 11 '21

Selenium, Chromdriver, HTML.

Learn how Web pages are built, and the rest is just tinkering with terminology e.g. What's a div tag, form input fields etc, and how to find objects in source code.

3

u/call_me_mistress99 Jul 11 '21

What exactly do you mean with system? Can you give an example?

13

u/epicmindwarp Jul 11 '21

Imagine downloading a CSV from Bank X and uploading to Azure Blob Storage, to read from Azure SQL Server.

Bank X has no API, so building that data pipelines is key i.e. How to get the CSV, and how to autoload it in.

8

u/call_me_mistress99 Jul 11 '21

Thank you for your response. I haven't understood anything so please tell me if my research is correct.

A Comma Separated Values (CSV) file is a plain text file that contains a list of data. These files are often used for exchanging data between different applications. They mostly use the comma character to separate data, but sometimes use other characters, like semicolons.

E.g. Name,Email,Phone Number,Address

Bob Smith,[email protected],123-456-7890,123 Fake Street

The idea of CSV files is that you can export complex data from one application to a CSV file, and then import the data in that CSV file into another application.

Bank x is a bank that has no branches and everything is done online.

Azure Blob Storage is a service for storing large amounts of unstructured object data, such as text or binary data.

A server is a computer program/device that provides a service to another computer program and its user, also known as the client.

Azure SQL Server is Azure's Server programmed in programming language named SQL.

An interface a device or program enabling a user to communicate with a computer. E.g. a graphical user interface (GUI) is a type of user interface through which users interact with electronic devices.

When you use an API, you're using an interface that lets the program you write control or access a program somebody else wrote.

As an analogy, suppose your program is a robot and you want your robot to get into a car and drive across the city.

When you drive a car, do you care how it works inside? Do you care if it's a V6 or V8? That it's a traditional gas engine, a disel, or a hybrid? All of those things affect how the car works, but those aren't the interface. The interface of a car is the steering wheel, gearshift, and pedals - plus all of the other levers and controls to access all of the car's other functions.

So if you were hypothetically building a robot to control a car, you don't have to teach it how an internal combustion engine works. You have to teach it how to accelerate, brake, steer, and shift gears, by working those controls.

So now let's relate this back to a programming example.

Twitter has an API. That means you can write a program that accesses or controls Twitter. You don't have to know how Twitter works internally. You just have to know their API - the interface for programmers to interact with their service. In the case of Twitter, it lets you write a program to do things like access someone's most recent tweets, or log into your Twitter account and post a new Tweet.

Most of the major web services like Google, Facebook, Reddit, etc. all have APIs - you can use them to automate almost anything you would do manually.

A data pipeline is a set of actions that ingest raw data from different sources and move the data to a destination for storage and analysis.

Don't know what autoload is.

TLDR:

Imagine downloading a plain text file that contains a list of data. These files are often used for exchanging data between different applications.

We downloaded the plain text file from an online bank and want to upload it to a program for storing large amounts of unstructured object data. Something with servers.

Bank x doesn't have an interface which allows outsiders to play with code belonging to bank x. So we need to build a set of actions that ingest raw data from different sources and move the data to a destination for storage and analysis e.g. how to get csv and autoload.

8

u/epicmindwarp Jul 11 '21

we need to build a set of actions that ingest raw data from different sources and move the data to a destination for storage and analysis e.g. how to get csv and autoload.

That is an excellent TL;DR, and the rest of your comment is spot-on.

The value add someone likes me provides is

  • Experience i.e. I already know what will/won't work and what areas would benefit a company to focus on

  • Knowledge of technology - I don't know everything out there, but I would know enough to be able to go away and find something suitable for your needs

  • Intricate knowledge of IT workings - I can tell you from a cursory oversight if your goal is achievable based on my knowledge of technology and finance (in this instance)

  • Crucial: Will your human-explained workflow be replicated by a robot, and if not (most likely) how can it be achieved (using available architecture)?

It's all well and good to say "I want to price this bond", but you as a human know where to look and what to look out for - how do you teach that to a machine?

That's where I make the big £££.

3

u/call_me_mistress99 Jul 11 '21

And one more question. What is an autoload?

3

u/epicmindwarp Jul 11 '21

Just any mechanism that allows you to automatically load files.

0

u/[deleted] Jul 12 '21

[deleted]

1

u/call_me_mistress99 Jul 12 '21

I asked what exactly OP meant with system.

2

u/TamahaganeJidai Jul 12 '21

Thats pure magic, love it thou! Microprocessorprogramming in assembler? Nah, thats fun. Data pipelines? yeah.... nah thanks i dont practice the dark arts XD

3

u/Anowv Jul 11 '21

This sounds like a more advanced version of what I do.

Finally getting something automated and hearing positive feedback is so f—king rewarding lol

Almost worth the crushing defeat everyday

3

u/epicmindwarp Jul 11 '21

Yeah, this is a simple example of what I do.

I've automated complete manual tasks, thinks 3 hour tasks, into 5 minute scripts.

3

u/raymond5555 Jul 11 '21

What type of roles usually work on the automation of thee type of task? Any suggestions on job titles to look at? I'm more interested in automation than analytics

5

u/epicmindwarp Jul 11 '21

RAD Developer.

It's not as glamorous as it sounds. Too many people are clueless about what they want the system to do.


That's the difference between myself, a BA, and a Developer.

A BA is bogged down by bureaucracy, takes a year of meetings to do anything.

The end user has no idea what a Dev needs or is capable of, they can only tell you what they do today - which doesn't translate nicely into computer talk.

The Dev often lacks context to figure out what the best method is to deliver something, and simply understands the tech and not necessarily the data itself.

Where I sit, hybrid data Quant role, I know what data I need, and what data pipelines and specs I need to get it as efficiently as possible.

2

u/[deleted] Jul 11 '21

What are checks and validations? Or where can I read/see an example of what that means. I see an opportunity to do something like this at my current job but I only know something doesn’t work when something tries to use the “cleaned” data and it just doesn’t happen on time or I get an error when I manually go check it.

I have no idea how to write validations into the process instead of still needing to manually check stuff.

4

u/epicmindwarp Jul 11 '21

It's based on the data, but some general rules apply.

Basic stuff is row counts, and columns that shouldn't be missing, missing compulsory data in specific rows.

Other stuff requires knowledge of the data e.g. Coupon rates shouldn't be negative for a bond, portfolio weights should equal 100, day on day market movements of a fund, not relative 2 standard deviations of market movement etc.

3

u/[deleted] Jul 11 '21

Okay that all makes sense.

Is this something you program in python to check? Like a script to report back row counts and coupon rates or do people open the excel file and check?

Sorry for the most basic questions. I’m learning things as I discover things.

5

u/epicmindwarp Jul 11 '21

It can all be done in a pandas dataframe in python. A human need only intervene if something goes wrong - so program the script to run all the checks and report back a big green yes, or an Amber yellow. Human only need to take action if things go Amber.

Rule of thumb, if there's logic to it, then it's probably doable in Python.

1

u/letmelookitup Jul 11 '21

Yup the first level you can consider schema validation, and the second level would be your application/industry-specific business rules.

1

u/namnnumbr Jul 11 '21

Have you tried Great Expectations or other libraries with precisely this kind of thing in mind? Wondering how ease of use compares to just writing my own checks in Pandas (which I’m familiar with) vs picking up a new package and adding dependencies.

2

u/12Eerc Jul 11 '21

How do you grab CSV files from websites where you have to click through a few screens?

2

u/epicmindwarp Jul 11 '21

Selenium or beautiful soup. The latter if simple, or selenium to do logins etc.

1

u/12Eerc Jul 11 '21

Yes mainly logins, will research that a lot. Thanks for the quick reply.

1

u/shibbypwn Jul 11 '21

I'm in the same boat. Everyone in the business world wants a "single pane of glass" and being able to make unrelated systems talk and share a mutual data set is the way to get there.

I've historically done this with on-prem servers and a whole lot of scheduled tasks - but lately I've been branching into serverless, event-driven systems in AWS (Lambda and API Gateway, with a FastAPI application).

1

u/[deleted] Jul 11 '21

don't you just use google drive to move data between systems?

3

u/epicmindwarp Jul 11 '21

Ha.

APIs mainly, and CSVs, writing to memory etc.

2

u/[deleted] Jul 11 '21

lol. I was only joking. Thanks

4

u/epicmindwarp Jul 11 '21

It's more sharepoint/teams these days!

0

u/[deleted] Jul 11 '21

[deleted]

6

u/epicmindwarp Jul 11 '21

Absolutely.

Not as extensively used as you might think, but definitely useful.

25

u/mastermikeyboy Jul 11 '21

I write Rest APIs in Flask, write scripts, and data processing pipelines (Apache Spark). The APIs can be simple crud, or complex actions.

I started as the companies only dev, now we're starting to hire specific devs for each path.

6

u/Dexteroid Jul 11 '21

Nice, I am at the QA and Jenkins part of that pipeline. So creating regression framework then creating pipeline jobs which can be invoked by dev or qa to test something specific. Tests are written in python but Jenkin stuff I use groovy for it.

2

u/[deleted] Jul 11 '21

Have you checked out fastAPI?

I used to use flask heavily for many different projects but got turned on to fastAPI about a year ago and much prefer it for REST apis now. It’s partially inspired by flask so it doesn’t feel like learning a whole new framework either.

4

u/mastermikeyboy Jul 11 '21

Yeah I don't like it :) mostly I don't like Pydantic. Marshmallow fits our use cases much better since it's allows true custom serialization. Combined with Flask-Smorest you get all the same benefits as fastAPI except for asyncio, which a library Flama solves.

Flama builds upon Starlette like fastAPI, but I found the creator to be too pure. So I forked it add convenience functions like fastAPI.

I've only done one small asyncio project with it thus far though.

1

u/namnnumbr Jul 11 '21

Those names sound like nonsense and also like I have research to do. Bookmarking for work tomorrow.

1

u/[deleted] Jul 11 '21

I'm writing my first simple API with Python. Do I need to use a framework for it?

I'd also like to use SQL in this project

1

u/mastermikeyboy Jul 11 '21

Depends on your requirements. If it's a hobby project, you should keep it light on libraries to prevent yourself from getting overwhelmed. I'd use Flask and Psycopg2.

If you need documentation and data validation, I'd use Marshmallow and Flask-Smorest.

If you want to use an ORM instead of raw SQL queries, then I'd use flask-sqlalchemy.

1

u/[deleted] Jul 11 '21

Thank you. It's a 'hobby project' but it will be useful to me and other people

1

u/__developer__ Jul 11 '21

I'd also recommend flask for a hobby project. It has been around for a long time and there are plenty of examples that can be found to help get your project working quickly. Its great for prototyping simple web app backends and APIs.

1

u/[deleted] Jul 11 '21

Thank you

19

u/Grouchy_Simple_1902 Jul 11 '21

I automate the testing of medical equipment.

  1. Take a manual test (example: Turn device on, open valve #69, check device reaction)

  2. Then write a python script that does the same.

  3. Commit to git

  4. review it with a colleague

3

u/raymond5555 Jul 11 '21

What type of title is this kind of work? For somebody more entry level? I want to do automation type work but not sure where to even look

13

u/Grouchy_Simple_1902 Jul 11 '21

When i look up jobs on linked in i usually go for "QA automation engineer".

This job (in my case) requires less fancy programing skills and more logical thinking like - How do i do X and observe Y at the same time if the device works like ABC.

I started working at entry level without even having knowledge of python (is that some sort of snake?). The company that hired me was nice and basically taught me how to do the job they hired me for.

Now i have an unrealistic expectation that other companies will teach me how to do the job they hire me for.

33

u/wsppan Jul 11 '21

Not python specific but here are things you will experience on a daily basis:

  1. Meetings. Both with peers and direct reports and management.

  2. Debug sessions with senior engineers looking over your shoulder (or you looking ober their shoulder)

  3. Code review sessions (formal and informal)

  4. Solo coding, debugging, documentation, running tests, CI, etc..

  5. Shooting the shit and bonding with co-workers

  6. Grabbing lunch (solo and with peers)

  7. Building, Configuring, tweaking tools, editors, IDEs, etc..

  8. Studying, researching, learning, exploring, etc..

8

u/[deleted] Jul 11 '21

This was my experience when I was last employed as a python dev

8

u/Dexteroid Jul 11 '21

I identify the test as the feature is being developed, I gather data and write tests to automatically test the feature. 99 percent it is done in python since we use pytest.

8

u/__developer__ Jul 11 '21

At my job I work on cloud based micro-services that bridge my company's API to external software APIs.

I've helped build our codebase from the ground up and its probably 90% python with the rest being Go, JavaScript, CSS, and HTML.
We use Agile/SCRUM development methods, so we have a two week sprint where all the devs know what tasks they'll be working on for those two weeks, then at the end of the two weeks we review the ending sprint, plan for the next, and then start the next two week sprint.

An average day usually consists of:

  • Daily Stand-up meeting; We all answer these questions:

    • What I did yesterday.

    • What I'm doing today.

    • Is anything blocking me from completing my work.

  • Work on assigned tasks

  • Work on bug tickets created by QA; if finished with tasks

  • Conduct code review and review pull requests

  • Meeting with product owner, QA manager, and dev manager.

There are also more meetings throughout the sprint and long meetings at the end/beginning of a sprint.

On average my job consists of about 50% programming, 25% code review, and 25% meetings.

6

u/king_booker Jul 11 '21

Probably late to this but here it goes (This is what I did last week)

  1. Created a pyspark job to read data from a table and push it into an ML algorithm (Written in Python)and write the output back into a parquet file

  2. Wrote a python script to copy files from a kubernetes container to another kubernetes container

  3. CI/CD pipeline creation for the job using jenkins

  4. Wrote tests for the project

I think what you learn with python stays but if you are working in python apart from coding skills you should be familiar with flake, isort for code quality

Know how to make a package, we use something called tox which is helpful

Know how to write tests for your code

Know how to make a jenkins pipeline to test your code in CI/CD ( I think a lot of teams have their dedicated jenkins people but its always good to know)

5

u/[deleted] Jul 11 '21

I'm a Web developer currently but a large portion of my job is data mining and webscraping using python. I'm scanning websites for errors, outdated plugins, mobile responsiveness etc.

I just turn up in casual clothes, put music on and work on scanning stuff. Quite fun :)

3

u/[deleted] Jul 11 '21 edited Jul 12 '21

Most of my work involved integrating features of open source applications built on Python ( like Mayan, Airflow, Superset etc.) I also did a lot of work in building pipelines on Airflow and web services with Django REST, Posthog (with JS) to track metrics for out clients. Most of our work is done in Docker, and we use GitLab as our repo and for CICD

3

u/skellious Jul 11 '21

I make prototype / "good enough" bespoke systems and hardware solutions for companies. I'm a single person company and I don't have any qualifications.

2

u/epicmindwarp Jul 12 '21

I started doing this about 3 years ago as part of a consultancy side gig.

Any particular area of focus?

1

u/skellious Jul 12 '21

Every job is a bit different, but generally they all have some aspect of "we need to connect this thing to that thing but we can't" or "there is no commercial solution for this and we need one but we can't afford two years and a 20 person team"

I don't charge as much as I probably should, but I'm still learning a lot of the tech I'm using so I feel that's a good compromise. Most recent job was at an activity centre making a timing system that linked to a database and to the equipment in use to track players scores/times.

1

u/epicmindwarp Jul 12 '21

Looks like we do very similar things!

I build a lot of bespoke builds for businesses that need systems to speak to each other - started out at non-profits.

1

u/skellious Jul 12 '21

Mine is similar but I also do some hardware stuff. Also currently working with a school to develop a low cost pollution sensor network for the kids to assemble and manage using raspberry pico and various sensors. It was my first time using a lot of the protocols like I2 C so that was fun.

3

u/MikeDoesEverything From zero to Data Engineer: https://tinyurl.com/2p8wd6y7 Jul 11 '21 edited Jul 11 '21

I automate data collection and processing much like the top comment by /u/epicmindwarp:

  • Get brought into meetings to discuss a problem.

  • Have a meeting after the meeting with my manager/team to discuss the best solution and/or what can be done.

  • Develop the solution and get it to whoever asked for it.

  • Help transition from on prem to serverless.

  • Also learn about new technologies and weigh in if they'd be useful or not.

2

u/raymond5555 Jul 11 '21

What kind of job titles are these descriptions? I enjoy this type of work. I do it manually with excel but want to develop this into a career type thing.

Any recommendations on entry level scope/title/work for these?

2

u/MikeDoesEverything From zero to Data Engineer: https://tinyurl.com/2p8wd6y7 Jul 11 '21

What kind of job titles are these descriptions? I enjoy this type of work. I do it manually with excel but want to develop this into a career type thing.

I'm personally a Data Engineer which is a very broad term for somebody who automates moving data about using code of sorts and maintains databases.

Any recommendations on entry level scope/title/work for these?

There is currently a shortage of DEs which means initiative and a willingness to learn goes a really long way so any DE projects would be great.

I'd research how to encompass the very basics of DE which is Extract, Transform, Load (ETL), SQL, and how relational databases work. After that, move onto cloud services and transition from local execution to serverless.

I started out by pretty much accidentally imagining my own projects as ETL pipelines and eventually discovered DE.

Without trying to sell you a dream, I actually love my job to the point where I think I might have a problem. If you like working "in between" teams and rather with projects (pipelines, processes etc) rather than dealing with people, DE is likely for you.

1

u/raymond5555 Jul 11 '21

It sounds perfect for me. I prefer project as there usually an end date and starting something new. Repetition is something I hate.

Do you do high repetition work at all? Or most of the project one time and done?

1

u/MikeDoesEverything From zero to Data Engineer: https://tinyurl.com/2p8wd6y7 Jul 11 '21

Do you do high repetition work at all? Or most of the project one time and done?

Building pipelines is, arguably, repetitive as you're really doing the same steps of pull data, save it somewhere, load it somewhere, add some views onto it, and automate it. That being said, I actually really enjoy it and work hasn't felt like work for a day since I started 4 months ago.

For my current role, we do 2 week sprints in an Agile team so I get something mostly new stuff every 2 weeks with some bigger projects taking longer than 2.

1

u/raymond5555 Jul 11 '21

It sounds perfect for me. I prefer project as there usually an end date and starting something new. Repetition is something I hate.

Do you do high repetition work at all? Or most of the project one time and done?

3

u/AchillesDev Jul 11 '21

I build the internal data infrastructure for my company. This is everything from our internal data architecture to web applications that use it to services that pull data from where it needs to be, puts it where it needs to go, processes it, automated the training and deployment of machine learning models, manages customer environments, etc.

3

u/Merakel Jul 11 '21

Lately I've been building a framework for our non-programmer (but still technical) team members to fairly easily implement API endpoints for automation.

3

u/tangerinelion (C++ Software Eng.) Jul 11 '21

As a C++ (lead) software engineer, I can tell you about that. I'll also use C#, C++/CLI, and Python depending on what I need to do.

We work in two week sprints, so there are some basic scrum stuff that we have - daily standups, a sprint demo, retrospective, sprint planning, backlog refinement. That ends up being about 8 hours per sprint. There are other meetings that I have with scrum masters as I'm also the team scrum master; that's an hour per sprint. We also have a coding standards team which meets one hour per sprint, so we can bring up any issues and publish best practices for our repository.

I'll probably spend about half a day per sprint working on customer-reported issues. Trying to figure out why something doesn't work and if there's anything they can do to resolve the issue; if not, I'll confirm it's a bug and then we'll schedule that to be worked on.

There's also an internal talk every sprint, so we'll spend an hour or two hearing from some other team about their work at a technical level.

Depending on the project and the state of things, I might have one or two meetings per sprint to discuss some software architectural issues or it could be more like one or two meetings per day. Each meeting is probably an hour.

I'll often have a task during a sprint to design something at a high-level, so the end result of that is to file more JIRA tasks. Simply filing the tasks can take an hour or more.

That all aside, in the times in between, I'll be pinged randomly on Slack for ad-hoc requests. Someone wants to know more about how some functionality I wrote works or has questions about the project that I know. Similarly, I'll be working on things and need to ping others for similar questions. I have a (non-lead) software engineer on the team who I'm in fairly close communication with as they'll run into unexpected issues and ask if I've seen it before.

Another great way I spend my time is doing code reviews. I'll often leave 10 comments on a simple pull request, and if I have the time to do an in-depth review of a substantial pull request I can easily generate 30 comments - each one indicating some kind of error-prone code, misunderstanding of a subtlety, or a performance issue. Similarly, when I have code in a pull request I'll be responding to comments.

In between all of that I am able to actually design and write code. Then compile it. Then test it manually. Then test it against our test cases. Assuming all of that works and goes as expected, then I can submit a pull request.

As far as "just typing on a computer" - yes, that's a lot of it. Lots of email, lots of Slack, lots of JIRA, lots of coding. All of it is typing on a computer. Lots of Zoom meetings.

3

u/CoffeeBaconAddict Jul 12 '21

There are so many good concrete examples of what people do and I hope you understand that almost everybody’s job could be completely different.

My job is to ensure data governance, documentation, fix bugs, code review, and support data scientists in building models by upgrading our code and data quality so they can use it.

I am the guy who has to reject 10% to 70% of the code that is committed because it doesn’t follow quality standards.

Never fails to surprise me that people who went to top-tier schools almost never seem to want to write tests.

So I have to reject their code and tell them that they need at least 3 to 5 tests for every function that they create.

So inevitably there’s some whiny, crybaby, bitchy complaint that they have they mutter under their breath, or the brave ones say it out loud.

I have a variety of phrases that whenever a developer or anybody really at our shop or at one of our clients says I purposely reject more of his or her commits because there’s a laziest programmers.

This is a little bit malicious and mean but the more often they say this I force them to generate more tests.

  1. Works on my machine

  2. Sucks for you

  3. Not my problem

  4. That sounds like a you problem

  5. I don’t need tests I write good code

  6. Well I didn’t change anything (says the person who literally submitted a commit)

  7. It worked yesterday

  8. It’s OK will fix that later

  9. I’m pretty sure I fixed that last week

  10. And the worst one of all time to say in a devil ops environment and I give special punishment for this “ maybe it’s your cookies you should clean those out and reboot”

So my current job is somewhere between ensuring that their code is much higher quality than they’re producing because we have regulations and SLA targets to meet. Code has to be easily readable, and also must be documented outside of code so it’s kept up-to-date.

3

u/xiipaoc Jul 12 '21

Not a Python dev (though there is some Python in our toolchain at work that I interact with occasionally). Basically:

First, we have our morning standup. Now that we're all remote, this is just a Google Meet meeting, about 10 minutes or so, where we briefly describe what we did yesterday and what we will do today. On Mondays we have a planning meeting instead, where we talk about the new things we as a team are going to do that week and estimate how much work they will take. This can take upwards of an hour, but if we don't have much new stuff to discuss (which happens when we're all already busy on big tasks and everything that needs to be talked about was discussed in previous weeks) it might be 15 minutes.

Rest of the day, mostly coding. There will be a break for lunch. Sometimes there will be meetings. When we do a new thing that we're not sure how we'll approach, we'll have a planning meeting, where we brainstorm the issues and come up with a plan for going forward. A few times a month there'll be 1x1 meetings, where we basically just catch up with each other and discuss whatever topics we want. When we were at the office, these were lunch meetings, so we'd go to a restaurant or something. 1x1 meetings with the manager or other authority figures on the team are when we discuss work performance and any issues that may come up with the team. As a non-lead, I may have two or three meetings per week, sometimes more but not too much more. If I'm starting a new project that falls under the auspices of someone on our business team (which is pretty new for us), we may have a kickoff meeting where we discuss the requirements.

Throughout the day, while coding, we will Slack each other questions and answers and generally have conversations. If someone's having trouble with something, we may do a screenshare and do some pair programming.

On Fridays, we have our company meeting and our team retrospective. The company meeting is when the business team gets to tell us all the stuff they did, the clients they made contact with, the newsletters they published, etc., and we get to demo the shiny new field we added in that one section of our app that we spent weeks doing back-end work for. Fun times. Retro is usually an hour long and we talk about the bugs we dealt with in the past week (and how to avoid them in the future), discuss what went well and what went not so well during the week, and choose an action item from the discussion to commit to implementing the following week.

Other than these, the vast majority of my time at work is spent actually coding. In addition to copy/pasting actually typing code into my IDE, coding includes looking up documentation when needed, debugging stuff that doesn't work, reviewing other people's code, Redditing while waiting for the app to start up, doing logistical tasks like creating users or writing database migrations or whatever, and basically everything I need to get the code from its original state to the state that passes code review having fulfilled the requirements of the task. I can't give you a good breakdown of how much time I spend on what here, because it definitely varies. The last few weeks I've spent a whole lot more time writing new code than anything else because we've had a few big client deadlines that I was doing most of the work on. Was? I mean am. We're not done. Other weeks I don't have a major thing to do so I pick off the bugs in the sprint or small improvement stories and do smaller things. Those weeks involve a lot of stepping through debuggers to figure out exactly why the thing over here is failing -- it's failing because it's getting bad numbers, but the numbers come from elsewhere in the call stack entirely, so it's a hunt and a half most of the time. Isn't debugging fun? Yes it is. I actually really like it.

Anyway, that's pretty much what I do. But, again, I'm not a Python dev; the code I work on does have a little Python in it, though!

2

u/imperial_squirrel Jul 11 '21

i program data and image conversions from one system to another.

most of my work is done in sql; but on occasion i need to change image formats, or read in a really messy report from an archaic system - that's when i use python.

2

u/chesquikmilk Jul 11 '21

I do alot of small database applications, usually with a basic interface sometimes an api. 80% of my time is spent designing in UML or meeting with steakholders and hammering down technical requirements. But all that design work leads to single week implementations and no surprises.

3

u/oakteaphone Jul 11 '21

meeting with steakholders

Must be delicious!

2

u/chesquikmilk Jul 12 '21

Thanks kind humorous stranger. Happy cake day xD

2

u/SomethingWillekeurig Jul 11 '21

I work as a data scientist in Python. So not a real programmer.

Not coding time: -meetings -helping BI's with logical thinking and interpreting data

Coding time: -web scrapers -data pipelines -data to csv's/insights for clients -creating machine learning models

2

u/Shnorkylutyun Jul 12 '21

Swearing at other people's code Swearing at my own code Swearing at the linter who adds squiggly lines everywhere

Some SQL Some REST APIs

2

u/jiavlb Jul 12 '21

I work a lot with django and django rest framework to build applications that expose Rest APIs. I use a lot of pytest as well (I love writing unit test cases with it). I would also like to add that I hardly get to write code from scratch. Most of the times the requirement is such that i have to modify existing code. So a lot of time goes in understanding how the existing code is written, what conventions and methodologies are being used. I also set up time to set up project to be run locally using vscode so that I can run the code and test cases locally and put breakpoints for inspection.

2

u/coaaal Jul 12 '21

Every day a team utilizes the various software tools I have created to generate files for printing and apply parameters to those print files to get it as close to the expected product as possible. This involves understanding the needs of various machines that use the files, and keeping to a good color profile workflow.

To organize the data from beginning to end I use a python client created with Tkinter. This allows the user to select input parameters and mass asset gathering via REST api that hits our asset management service. I also have some Trello REST functionality to work around a poorly thought out workflow established by one of the departments. Sometimes there is a "toss shit against the wall and see if it sticks" mentality when departments come up with new management systems. I enjoy having unique problems to solve to work around these poorly designed systems so I dont complain too much. Job security....

We are also in a alpha stage of testing some new software I pushed at the product design level that will allow us to bypass the need to manually output files at all. Once the user is done creating the product design, all of the puzzle pieces are available to create the product without it having to go through another department, rather a service that another team is developing.

Besides upkeeping these projects and adding new feature requests, I am always looking at ways to automate workflows for other departments. I've been with the same company for 7 years and started at a real basic non-coding position. I am very grateful for the opportunity I was given and understand that I was at the right place at the right time with the right interests that pushed me to learn what I needed to grow. And I am a very lazy person...

2

u/ShivohumShivohum Jul 12 '21

Thanks OP for this post. I Kind of needed it too

1

u/[deleted] Jul 11 '21

I convert Alteryx workflows into Python scripts...

1

u/MannyManMoin Jul 17 '21

Im a software developer that went from assembly in the mid 80s to visual basic/.Net beginning of 2000s, back to C++ mid 2000s, to developer manager 2010 to small business developer in 2018 writing raspberry pi python.

Now transitioned to Flutter/dart as that makes me able to write software for both Apple silicon and Android. And with some of the most popular libraries from Python being ported to flutter libs, Im not gonna go back to purely python moving forward. Its a love hate relationship.

Why python ? well its easy programmable and prototypable, both in machine learning, and micro controller software. With Raspberry and Python you can prototype stuff for small business and just make money out of it. With Flutter, you can design apps for both mobile platforms with same shared codebase for the raspberry microchips. Also python lets you program Risc5 microchips as micropython which is awesome, dropping linux entirely on microcontroller part.

Python will stay with me probably next 10 years aswell as python has stayed with me since beginning of 2000s. But I know very well with new arm chips like apple Mx chips and soon Intel/microsoft/amd chips being more efficient for mobile market, python will be less important and we will transition over to compiled code to be more efficient on the new platforms. This is where flutter comes in.

If you ask me as in 2021, learn flutter..

Google is pushing this platform hard, as it can compile code for different platforms sharing the codebase.

0

u/[deleted] Jul 11 '21

[deleted]

2

u/AALLI_aki Jul 11 '21

Thanks for the reply this was very informative

0

u/[deleted] Jul 11 '21

[deleted]

1

u/AALLI_aki Jul 11 '21

Are you employed? What packages you Use?what do you achieve at work? I know they type but I meant what they write

0

u/[deleted] Jul 11 '21

[deleted]

1

u/AALLI_aki Jul 11 '21

Doesn't that make u a python programmer?