r/Python • • Apr 18 '24

Discussion Why Jupyter notebook

I did a search and read previous discussions in this sub about why Jupyter labs exists. From what I understand it is for quick code to try things out especially in data science …. Thing is, can’t you get the same thing via a decent IDE with autocomplete and quality debug tools in some quick Python code like in VSCODE? I do that all the time….

I develop RNA computational software in Python and I abandoned Jupyter labs very early on as it just was not well suited to the job of bespoke data science. So many limitations and don’t get me started with widgets… I’ve been developing code for decades and I have spent 3 days trying to understand widgets and I think I finally understand how to modify a text box and get a value from it…. You have to observe (why?), then act on, then route through some output and then you can get the info? It’s like someone was drunk while trying to emulate visual C#.

Edit: before anyone else tells me to read vscode docs like I’m an idiot… I am unable to remote into the repo I’m using now for reasons. Not everyone can remote in and the question is not about remoting in. It’s still garbage in VSCODE… just use regular python.

Edit2: for everyone who says they use it to do chunks of code work… I developed my own data management system for that kind of stuff that provides a wrapper over serialization that essentially creates pythonic pointers to data on any network drive. I just grab my data that I want to grab. Provides autocomplete and C# like attributes. I can dynamically build any data storage class using UML formatted infrastructure-as-code to build the front end. The back end is completely dynamic. Here is the pypi link. Note that I developed this to handle genomic data that was generated over the course of a week of 24/7 computation.

https://pypi.org/project/data-nut-squirrel/

0 Upvotes

70 comments sorted by

24

u/jjrreett Apr 18 '24

I use jupyter like a repl. It’s great for problems that have a major IO operation then lots of exploration to figure out how to process and visualize. Plus the —to-html/pdf functionality is great. Makes making reports super easy.

For what jupyter is good at, there really isn’t anything like it.

1

u/IllogicalLunarBear Apr 19 '24 edited Apr 19 '24

I developed my own data management tool for that so that I can break my genomic software into multiple chunks and I just grab my Python classes that are populated that I store on a remote drive and access as needed via a dynamic back end that handles addressing and data management that is accessed via a Python module that you build via infrastructure-as-code that is UML formated and stored the classes as yaml files or pickle if you so like.

Then at the end of 4 different scrips running for weeks I can grab the data I want and plot as needed on the fly… it can be weeks later after the data is generated and after multiple power cycles. It can even pick up where you left off if the scripts that build it crash in the middle. I don’t think Jupyter can do that…

18

u/KingsmanVince pip install girlfriend Apr 18 '24

You can use jupyter notebooks in vscode and still get autocomplete.

Besides reading previous discussion, I suggest you read the docs

https://code.visualstudio.com/docs/datascience/jupyter-notebooks

-13

u/IllogicalLunarBear Apr 18 '24

Not with the current project I’m doing for work. I’m am unable to remote into it due to reasons. I have to do everything in Jupyter right now and it’s painful.

7

u/joaofelipenp Apr 18 '24

You can tell vscode to use another jupyter server (remote or local)

4

u/andrewaa Apr 18 '24

do you know that you can call functions in jupyter?

you develop anything you want in any platforms as you like, and then put them into functions in some files and upload to your sever folder and then in jupyter call those functions.

if you really don't like it but you have to use it, why don't you just come up with some workarounds to make it work?

1

u/iMDu093 Apr 18 '24

Trust me, I get it. I’m forced to work in a notebook environment as well. And believe me, there are worse ones out there than Jupyter. While you question why Jupyter lab/notebook exists, I question why paid services that do pretty much the same thing exist when companies could just use Jupyter for free haha. But I started to realize more and more that the “best” tool may not be the best in every situation.

You will get more frustrated later I think. Because if your pidgeoned into using Jupyter I’m guessing a good amount of folks u work with arent software engineers. You probably will see some really bad coding practice. Almost no one will know what PEP8 is, let alone have read it. Global variables everywhere. Naming conflicts cuz ppl are too lazy to write module.blahblah

I’m always don’t to vent about this hahah. But honestly it’s not the tool that’s bad. And I don’t think we should be saying the folks who work on Jupyter were drunk, they have other jobs, and it’s amazing that we have a free open source tool like Jupyter.

9

u/sirjackholland Apr 18 '24

Why are you trying to set up widgets when you're just learning how to use notebooks? Declare the variables at the top and call it a day. Then, when you're more comfortable with notebooks, you can make them fancier by adding widgets if doing so actually improves the functionality (many notebooks have no need for them).

-8

u/IllogicalLunarBear Apr 18 '24

I’ve used notebooks in the past and I understand them. I need to make the code easy for people who are not developers to use.. this the widget to handle user input.

1

u/sanbales Apr 18 '24

I'm honestly surprised you are having a hard time with widgets. Widgets are not hard at the beginning. The trouble starts when you make a complicated UI and don't think it through, then you end up with an overcomplicated state space that's all tangled up. IMO when you architect your widgets correctly, they can be quiet elegant. What I like about widgets is that they mercilessly punish me when I don't architect things the right way... but maybe I'm a glutton for punishment.

I'll admit that it takes time to get there, and probably more than you will feel it should once you get it, but once you do, you'll realize the choices they made were not bad.

W.r.t. observers, you have other options, although that it the most general way of integrating widgets, but sometimes a link or dlink is a better way. There are honestly only a handful of things to learn about widgets.

2

u/IllogicalLunarBear Apr 19 '24

I realized that I was making it harder than it should r and was working off someone else’s script. I decided to just do what made sense and it worked… lol

6

u/BranchLatter4294 Apr 18 '24

If you don't have a use case for notebooks, don't use them. Many people find them helpful. And it's not just for Python. VS Code has good support for notebooks in other languages as well.

-5

u/IllogicalLunarBear Apr 18 '24

I have to use it right now for my job and I’m working in a 3rd party Jupyter notebook accessed via a weird portal.

9

u/nemom Apr 18 '24

I don't use VScode so I don't know if it works the same. Jupyter is interactive. You don't just run a program from start to finish and exit, you run the cells step-by-step. All results are available for further computations... If you have a large data file, you can read it in once, then explore it. If you make a mistake, you just edit the cell and re-run it. When you have the steps all worked out, you can export to a stand-alone program to run anytime you want.

6

u/Spin-Stabilized Apr 18 '24

This. This is exactly where it has been fantastic for me. I have notebooks for analysis of large sets of telemetry data. On the order of a gigabyte per day. If I want just a couple days worth of data, it takes a couple minutes to read it into a data frame. If I want the whole set since launch, you’re talking a couple hours. With Jupyter I read it in once and now that it is available to further cells I can modify my code without waiting for data to be read in again. I can quickly share plots out of the data and go back for quick manipulations based on feedback.

If I then find that I need to run the same code on a regular basis, I port it out to a python file, wrap it with a script, and I’m all set.

I think of it like Matlab for other languages.

1

u/IllogicalLunarBear Apr 19 '24

I have the same issue as I generate massive amounts of genomic data and I use my own data management tool that stores my classes as files on a remote drive that I can access very quickly. It allows me to break up large chunks of analysis and then I can plot or do manipulation of the data later when o get around to it. I’m running analytics on data I generated months ago right now that I am accessing like it is any other normal Python class.

-2

u/Training_Butterfly70 Apr 18 '24

An IDE does all this better. The point of Jupyter is presentation. Analysis, quick charts, tests, and presentation is what Jupyter is good for. Terrible practice if you write production ready code in there and then just export.

2

u/davemoedee Apr 18 '24

An IDE isn’t an alternative to Jupyter notebooks. An IDE is where you edit Jupyter notebooks.

IDEs can be used for debugging, but IDEs aren’t going to retain state while allowing you to easily run arbitrary code. A python shell can. A notebook can.

Notebooks are great for iterative exploration.

1

u/IllogicalLunarBear Apr 19 '24

I developed an application that retains states between power cycles that works best with quality IDE that allows autocomplete of the archived classs.

1

u/Training_Butterfly70 Apr 18 '24

You literally just said what I said in a different way 😂 not sure what the point my downvotes. For building a production ready application, use an idea. For exploratory stuff and presenting findings to stakeholders (stuff like that), it's great to use Jupyter notebooks.

1

u/runawayasfastasucan Apr 18 '24

No he didnt, you said that an IDE does this "better". Since he was talking about notebooks it is normal to assume that this is what you meant.

5

u/LaOnionLaUnion Apr 18 '24

Imagine you want to publish something like a scientific paper but have it run the code interactively against real data. That’s pretty much why I think it was invented. It’s great for some things but when you really want to develop and run software it’s very limiting

-1

u/IllogicalLunarBear Apr 18 '24

I mean I run my code against real data….

4

u/LaOnionLaUnion Apr 18 '24

Sure but are you writing reports for other people? I think that was the key point that people reading a paper could look at real code and run it against real data. As opposed to static charts

3

u/IllogicalLunarBear Apr 18 '24

I’m starting to understand this use case better. Thanks

168

u/RedMaskedMuse Apr 18 '24

My use cases for Jupyter are:

  1. executable documentation (show how an algorithm works piece-by-piece, with markdown sections in between code sections; also works for lab reports with post-processing)

  2. rapid prototyping (quickly iterate on pieces of an algorithm without having to re-execute the rest of the code - especially important if some of the steps are slow)

  3. tutorials (allows the student to work through code piece-by-piece while being able to see text, image, or plot results in between sections)

It isn't the right tool for everything, but it does come in handy for some tasks. It's a partner to an IDE, not a replacement.

11

u/SmegHead86 Apr 18 '24

Perfect answer, and let me also just add that as someone who works in consulting, the documentation is key and I've been fighting to make notebooks the norm in our scripting process so that we can break things down into steps or blocks of code that are always well documented and repeatable.

A lot of what we do is one and done migration work, so having a notebook kills two birds (code and documentation) with one stone. Can I document things in a script, yes but markdown looks soooo much better.

1

u/mason_savoy71 Apr 18 '24

I use them exclusively as tutorials and reminders to myself of what I've written.

0

u/BerriesAndMe Apr 18 '24

Is there a way to just show the code in jupyter nowadays? Always drove me made that you have basically no chance of seeing the workflow in the code because everything is interrupted and segregated..plus you never quite know which version of which field is currently loaded and executing so you need to restart the entire kernel for most runs anyways.

12

u/theSpaceMage Apr 18 '24

You can convert a Jupyter notebook to a Python script, although, the markdown becomes comments in the code by default. You can convert to a script and remove the markdown with:

nbconvert --to python --RegexRemovePreprocessor.patterns="^%" notebook.ipynb

It's annoying to need to use regex, but you can just create an alias for it in your shell of choice.

You can also look into the jupytext package which I think provides a simpler way to do it.

0

u/BerriesAndMe Apr 18 '24

Ah I was thinking of 'hiding' it temporarily. I sometimes get jupyter notebooks to 'review' and it's just a pain because the code is so cut up. However, the eventual purpose is to submit the notebook itself to wherever so i can't turn the notebook into plain code (even if I'd like to).

I know you can hide all the code so that it just renders the pictures from the code, I was hoping the opposite of just showing the code and hiding all the markdown & output would also be avialable.

1

u/theSpaceMage Apr 18 '24

Oh I see. Yeah. That's a bit annoying. There's an extension to make markdown cells collapsible, but still requires manually collapsing each one.

1

u/BerriesAndMe Apr 18 '24

Haha, ok. I'll look into that. In the mean time I'll just continue to complain about how jupyter is the absolute worst while knowing basically nothing about it. Lol

2

u/kernco Apr 18 '24

I use Jupyter notebook daily for bioinformatics. I don't know what those widgets you're talking about are and what you mean about getting a value from a text box. That's not a feature I use and I've never really seen anyone else use. The reasons I use Jupyter notebook are:

  1. Code can be easily tested. It's very quick to implement a small piece of code, see the result or the error it gives, make changes, and re-run until it's working. Then it can be copied into a much larger script with more confidence that it works.

  2. When I make plots, I see them instantly. I don't have to save them to PNG files and open them. This again makes it much quicker to tweak the different details of the plots and get it to look how I want.

  3. Data can be explored more easily. I can load it into a pandas dataframe or even just a list or dictionary and then look through the data in different ways.

  4. My analysis can be shared with others, with notes and plots integrated into the code used for analysis. See for example this.

You're right that #1,2,3 can also be accomplished with an IDE. I think the reason Jupyter has become so popular in data science is that a lot of data scientists don't have strong computer science/programming backgrounds. They have learned some programming as a sort of supplementary skill alongside biology, physics, etc. and they are not as comfortable or familiar with setting up and using IDEs and debug tools.

1

u/IllogicalLunarBear Apr 19 '24

Yeah, I provide independent consulting work to universities for RNA stuff since I’m a software engineer with 15 years of RNA experimental testing analysis experience. The scientists I work with write good functional code that does amazing things… it’s just not very maintainable…

37

u/iMDu093 Apr 18 '24

Sounds like you have a cool job. Yeah I don’t think Jupyter is the best choice for building software. Nobody would argue that. But it’s quite nice for EDA and creating reports/presentations. You can save your notebook as an html file, pdf, md, etc.

Why Jupyter? It’s like asking why would anyone have a desktop when you can carry a laptop. Different tools are for different things. Use whatever IDE you like, nobody is forcing you.

Sounds like you had a rough time trying to get some things going in Jupyter. And if you dislike it this much, I pray that you never have to touch a databricks notebook hahahaha

6

u/czar_el Apr 18 '24

This is the answer.

Jupyter is not intended as a software development tool or a computational model building tool. It is a communication tool. Rapid collaborative prototyping or EDA, presentations, audit trails, or tutorials, all aimed at non-expert audience greatly benefit from Jupiter's ability to mix markdown cells with code/output cells, as well as the ability to quickly share in many formats, some of which (like HTML) do not require the recipient to have Jupyter.

30

u/ConcreteExist Apr 18 '24

Jupyter Notebooks aren't typically for writing software, they're for doing data analysis and documenting your process as you go along. I wouldn't use one to do actual development work though.

5

u/joaofelipenp Apr 18 '24

You have to observe (why?)

You don't have to observe. If you have the widget variable, you can easily access its value. The observer is just a way to have a callback that you can pass so your code gets notified when something changes.

IMO, widgets in jupyter are among the easiest UI that you can do in Python

3

u/sanbales Apr 18 '24

I'm a little baffled they are having a hard time with ipywidgets...

4

u/Existing-Account8665 Apr 18 '24

I'm not in the target market either, but if I needed to send someone my REPL session, including pretty graphs, that someone non-technical could view in browser (instead of a professionally made piece of software I was proud of) then Jupyter might fit the bill.

1

u/Training_Butterfly70 Apr 18 '24

It's just good for prototyping, quick analysis, and presenting findings. For real software deployed in production use an IDE

1

u/IllogicalLunarBear Apr 18 '24

That makes sense… I guess it’s easier that spinning up matlibplots and saving the images for plots you want to show. I’m using it right now to just manipulate some data.

1

u/nickkon1 Apr 18 '24

So I got the task to model something or explore a dataset that came as an excel. Let's write a python script for that.

  • It loads but the headers begin in line 2. Quick fix, change the loading and rerun.

  • I take my target and put it into a linear regression and rerun.

  • Some errors appear because some things are strings instead of numbers? Change the code and rerun.

  • From my expertise I Know that multiplying A by B will have an awesome feature that has predictive power on my target. Do the calculation and rerun.

  • My model doesnt become better? Rerun with debugger and expect the coefficients.

  • Ah, I found something weird. Change it and rerun.

And now imagine each rerun taking 10mins.

Or, I simply do my explorative work in a Jupyter Notebook. Once I know what my whole pipeline looks like, I will create create a proper script.

1

u/IllogicalLunarBear Apr 18 '24

I mean I do that all the time in regular Python with pandas and whatnot. It’s not that hard

1

u/Choles2rol Apr 18 '24

I use Jupyter as a REPL basically, comes in handy for testing things out if I wanna not reload a whole script or program and riff on something with a big chunk of data.

2

u/IllogicalLunarBear Apr 18 '24 edited Apr 18 '24

I developed my own data management system for that kind of stuff that provides a wrapper over serialization that essentially creates pythonic pointers to data on any network drive. I just grab my data that I want to grab. Provides autocomplete and C# like attributes. I can dynamically build any data storage class using UML formatted infrastructure-as-code to build the front end. The back end is completely dynamic

https://pypi.org/project/data-nut-squirrel/

1

u/Choles2rol Apr 19 '24

Oooh will take a look at this, looks very cool!

1

u/IllogicalLunarBear Apr 19 '24

Thanks. Hope you enjoy. I developed it to handle the massive amounts of data I generate with my bespoke RNA analysis software.

1

u/TistelTech Apr 18 '24

fun fact, the idea is not new, Knuth (Stanford CS prof. of Art of Computer Programming fame) came up with it ~40 years ago.

https://en.wikipedia.org/wiki/Literate_programming

as others have said, you don't need to go back to the beginning, you stay at the coal face. Say your script starts off by downloading a whack of data then the poke around starts, you only need to download once, then keep working.

Pro tip if you face this problem and don't like the jupyter notebook interface. Check out the pickle lib:

https://docs.python.org/3/library/pickle.html#examples

basic idea:

does the pickle.bin exist? just pickle.load it. otherwise download the expensive/slow data, then pickle.dump it. Then when you change some code and want to run again, on the second and subsequent runs it will quickly read it from disk and skip the expensive/slow download and/or calcs. its a poor person's jupyter.

1

u/IllogicalLunarBear Apr 19 '24 edited Apr 19 '24

I’m quit familiar with pickle and my code is able to handle it if it’s added to routing code. I’m using yaml right now as I want to be able to read the text vs the binary data stream the pickle provides

Edit: my software was purpose written to handle genomic data that takes a week to compile… also my data-nut-squirrel code is not literate programming or even close

2

u/priestoferis Apr 18 '24

My current thinking is that it's great if say somebody is coming from from a business major and getting into data analysis with python. If the company has a jupyter installation they can quickly this person a remote, continously running and managed python environment, without needing any understanding of stuff like ssh and python environment management or any knowledge of how to install and set up frameworks that require a bit more than pip install.

It also has it's niche in presentation, but I do think it is very niche. Even my direct boss prefers to look at ppt-t, production people will need scripts and likely like simple git diffs for changes rather than forcing ipynb through hoops to get readable diffs. So it's mostly good for sharing between people who also work with notebooks.

2

u/Mammoth_Pool_2927 Apr 18 '24

We use JupyterLab with JupyterHub so multiple people can work realtime on remote server. Very nice project with security features for national bank.

2

u/tonnynerd Apr 18 '24

I do agree that jupyter is shit for anything "finished", with the notable exception of maybe as a runnable documentation.

However, it is amazing for prototyping and debugging (yes, debugging. Bear with me).

Pretty much everything I do nowadays starts with a throwaway jupyter notebook on VSCode, then once the logic and structure is more or less right, it goes to an actual module, and the calls I was using in the notebook became the first test cases, too.

For debugging, when it's not clear where/when things are going wrong, I like to import some pieces of the system and try to reproduce the problematic part in the notebook, taking things apart until I come to a minimal reproduction of the bug. If this doesn't lead to a solution, it will at least give me a small reproduction to use to ask for help.

So, yeah, it sucks as an API or entry point, but it is a great tool. I particularly like using it inside VSCode because I can then use pretty much all the same commands and shortcuts that I normally use on files in the cells.

1

u/[deleted] Apr 18 '24

I have used Jupyter notebooks every day for a couple years now and I have no idea what widgets you are referring to. Notebooks mostly shine though when you are either trying to share a repeatable process that input varies on, or when you are trying to sell a story with the analysis since you can intermingle markdown.

It’s also much cleaner and simpler for prototyping and exploration than standard IDE debugging is.

1

u/joaofelipenp Apr 18 '24

ipywidgets

I don't think notebooks shine when the input varies. I'd rather have a script that I can run with arguments instead of a notebook. Especially when the notebook is in development and I need to replicate the changes to multiple notebooks that run the analyses with different variables.

But they are great for telling a story, prototyping and exploration indeed

1

u/[deleted] Apr 18 '24

I tend to give them to members of my team who don’t know much python and the input conditions can be documented in the notebook so it makes it easier for them to

1

u/[deleted] Apr 18 '24

Jupyter notebook is good for exatcly the same thing as a real notebook: try stuff unorganizally.

You can test a new lib, a connection to an API/DB, open a dataset to see what the data looks like, make a quick analysis on a dataset and create a report, it's a playground !

But it's not for working with multiple people on it, not for writing a real feature, not for anything that will be reuse. Never use a notebook for production stuff, that's evil at it's finest.

1

u/MissingSnail Apr 18 '24

I mean, for anything serious you should have lsp and git plug-ins running in your Jupyter lab.

Remote development is just log into remote machine, run jupyter lab with —no-browser and whatever port tunnel or such you might need, then open the browser on local machine and connect.

1

u/mfb1274 Apr 18 '24

Jupyter notebooks in VSCode are the way to go

1

u/gxcells Apr 18 '24

You have to run a python script entirely each time. You cannot quickly try something with a python script without having to comment out code. With jupyter notebook you run what you want when you want, you do not need to run all code. You can test several things and keep the code of each test in a different cell . This is just amazing. I just wish the whole google colab stuff could be implemented in jupyter notebook (amazingly easy widgets, etc...). I

1

u/runawayasfastasucan Apr 18 '24

Just fyi, notebooks doesnt have to be Jupiter lab. 

-3

u/house_lite Apr 18 '24

Bc python sucks

2

u/[deleted] Apr 18 '24

[removed] — view removed comment

1

u/mmmmmmyles Apr 18 '24

this has been our inspiration for https://github.com/marimo-team/marimo