The official start time for the AMA will be 5:30pm UTC on March 2nd, before then this post will exist to collect questions in advance. Since most of us live all over North America and Europe, it's likely we'll answer questions before & after the official start time by a significant margin.
pandas is a Python package that provides fast, flexible, and expressive data structures designed to make working with "relational" or "labeled" data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language.
We will soon celebrate our 2.0 release. We released the release candidate for 2.0 last week, so the actual release is expected shortly, possibly next week. Please help us in testing that everything works through testing the rc :)
Ask us anything! Post your questions and upvote the ones you think are the most important and should get our replies.
- Patrick, on behalf of the team
Marc:
I'm Marc Garcia (username datapythonista), pandas core developer since 2018, and current release manager of the project. I work on pandas part time paid by the funds the project gets from grants and sponsors. And I'm also consultant, advising data teams on how to work more efficiently. I sometimes write about pandas and technical topics at my blog, and I speak at Python and open source conferences regularly. You can connect with me via LinkedIn, Twitter and Mastodon.
Marco:
I'm Marco, one of the devs from the AMA. I work on pandas as part of my job at Quansight, and live in the UK. I'm mostly interested in time-series-related stuff
Patrick:
I'm Patrick and part of the core team of pandas. Part of my daytime job allows me to contribute to pandas, I am based in Germany. I am currently mostly working on Copy-on-Write, a new feature in pandas 2.0. (check my blog-post or our new docs for more information).
Richard:
I work as a Data Scientist at 84.51 and am a core developer of pandas. I work mostly on groupby within pandas.
Last years has been better. pandas got some funding, including few core devs being paid to work in pandas in companies such as Quansight, Intel or NVIDIA. And we also received money from the Chan Zuckerberg Initiative, Tidelift, Bodo and smaller donors. Just few years ago funding was very limited, but today, we're lucky to be able to have a decent amount of paid maintainers.
What was the funding process like getting started? In my area of work (science research) it seems like funding only comes in for a project after you’ve done the majority of the project. Was there a plan for getting Pandas funded or did the project grow organically until you realized you could get funding for it?
As far as I know there was no/very limited funding for a long time. most of the work was done by volunteers only in the beginning. Over the last years this got a lot better though.
Anaconda was a company that hired developers to work on Open Source relatively early on.
For many years there was only the support of few companies letting people work on pandas as part of their job, and small personal donations via the NumFOCUS website. That money helped cover small expenses like CI services.
The main difference came with CZI, who started supporting open source software used in biology. We got funding to start paying for hours of maintainers with it. Also Tidelift provided monthly payments in exchange to implement small practices, like having a standard (and not customized) license, and providing a way to report security vulnerabilities. We got some other funding, and now more maintainers allowed to work on pandas as part of their job, but the situation is good mainly because of that particular funding. NumFOCUS provided some funding to for specific projects (with the money that comes from general NumFOCUS sponsors, and PyData conferences).
Why choose mm/dd/yyyy as default date rather than dd/mm/yyyy 🤔? (Just banter from an european guy) Real questions:
what are the main improvment focus going forward ?
what caused you the most problems / was the most complex parts during delevopment ?
what was the most fun / rewarding parts during development ?
in my work, I use pandas as a data processing engine (kinda), the data I process if often heterogeneous and full of holes / discrepancies, I often find myself finding with rhe way pandas handle errors as most of the time I just want to log the fact that this row had a error. Why not put a 'error' arg to apply, just as in astype and such ?
I also would like to thank you guys for your amazing work, pandas has been making my life easier everyday, you are really doing amazing work.
We are spending a lot of time on improving the extension array interface right now. Right now there are some parts that are special cased internally for our own extension arrays which makes it harder for third party authors to implement their own without falling back to NumPy. GroupBy is a good example for an area where we are still not as good as we would like. This becomes kind of necessary for improving support for our pyarrow extension arrays as well.
We have some areas in our code-base that are pretty complex, indexing is one of them for example. In general, we try to avoid breaking stuff in an incompatible way in minor releases. This makes improving pandas tricky sometimes, because it stands in the way of cleaning up internally/refactoring internally to be more compatible with new stuff.
in my work, I use pandas as a data processing engine (kinda), the data I process if often heterogeneous and full of holes / discrepancies, I often find myself finding with rhe way pandas handle errors as most of the time I just want to log the fact that this row had a error. Why not put a 'error' arg to apply, just as in astype and such ?
According to this blogpost by /u/datapythonista it sounds like a limitation of the numpy backend dataframes are built-on, check out this excerpt, I bolded the relevant part:
While NumPy has been good enough to make pandas the popular library it is, it was never built as a backend for dataframe libraries, and it has some important limitations. A couple of examples are the poor support for strings and the lack of missing values.
So maybe something we can hope to see fixed with the migration to Arrow in 2.0?
Yeah with NumPy you'd always end up with float when setting missing values into a integer array for example, this isn't the case any more with our own nullable dtypes and also with the arrow dtypes.
> Why choose mm/dd/yyyy as default date rather than dd/mm/yyyy
I presume you mean, when a date could be ambiguously read as either month-first or day-first? Like 02/01/2000.
In the past, pandas would prefer to parse with month-first, and then try day-first. Unfortunately, it would do so midway through parsing its input, because it was very lax about allowing mixed formats. This would regularly cause problems for anyone outside of the US (which I think is the only place in the world to use the month-first convention).
I honestly prefer ISO8601 format (YYYY-MM-DD). Both the ones you mention are ambiguous, and if I read 03/02/2023 I've no way of deducing which one is the month and which one is the day. The ISO standard is unambiguous.
Before I ask my question, I would like to really thank you for the amazing library that I use daily in my work.
That said, there's maybe one thing that is still bewildering to me:
Why are the APIs of read_csv() and to_csv() different?
For example, df = pd.read_csv(..., header=False) is not allowed, and I still stumble over it every other time. I'd understand if it meant something specific that is different to None, but this feels like it wouldn't be stepping on anything's toes. df.to_csv() accepts both.
And then, read_csv() will by default introduce an index that wasn't in the file, but will not introduce a novel header – it will use the one that's there. But to_csv() will write the file with the new index, but, of course, with the old header. Which means that if you do a single back and forth with the exact same kwargs, i.e., pd.read_csv(**kws).to_csv(**kws), you end up with an extra index column.
There must be some kind of a reason due to how things are structured internally. I think just knowing why it is the way it is will be enough for me – I'm not saying it has to be changed or anything.
Very good point. I myself find the index column in the output csv annoying every single time I use `to_csv`. I wasn't in the project when that was implemented, but I assume the reason is that pandas was initially implemented for financial data, and the index was mostly the timestamp and not the default autonumeric. If that was not the data pandas developers had in mind at that time, probably pandas wouldn't even have row indices (I think Vaex doesn't, not sure about Polars).
The next question is why we don't change it now. And it's something worth considering, and you're free to open an issue in GitHub. But in general, pandas developers (others much more than me), try to not break the API, unless it's in cases where very few users will be affected and the status quo is obviously inconsistent. I'd personally like to see that changed, but I don't think it'll be easy to get consensus.
What I think it can make sense is to try to move all pandas I/O (read_* and to_*) to third-party projects. In that case the pandas to_csv would continue to behave in the same way, but hopefully someone would develop a new one like to_csv(engine='whatever') that could potentially be faster, have a better API, and more appropriate for your needs. But let's see if there is consensus for this to happen.
An entire rewrite of the code behind apply / agg. Internally their code paths interweave in complex ways, and can be surprisingly slow is some cases. Depending on what object your on, the API is slightly different.
Cleaning this up and making it better while also making the gradual changes so as not to be disruptive to users is difficult, time consuming, and slow. But we're working on it!
I'd remove having a row index (at least by default), and the I/O API: being consistent with read_*/write_* or from_*/to_*. I'd also probably remove half of the code in pandas to other third-party extensions.
OK, thanks! IIRC, pandas was originally inspired by R `data.frame`s, so I figured the devs might keep a sharp eye on what's happening on the other side of the wall.
Pandas used to be the only kid on the block, but it seems there are some other libraries popping up claiming to be faster/better/etc. Have you evaluated any of these other libraries to potentially integrate features into pandas (or improve existing ones)?
Personally polars' strictness is making me think about situations when in pandas we end up with object dtype, which we should probably avoid. Here's an example: https://github.com/pandas-dev/pandas/issues/50887 (polars would just error in such a case, which I think is the correct thing to do)
I don't think that it is realistic short term to add out of memory support. Generally, I'd recommend going to Dask for this, it supports our API very well with bigger datasets.
Implementing something like lazy evaluation would be a major major breaking change on our side and hence not feasible right now
Being able to use Apache Arrow internally. I wrote an article with the details about it, since it's not trivial for regular users to understand why this is important.
I'll also mention copy-on-write. And I know it's not exciting, but all of the bug fixes throughout the code that make pandas more predictable and reliable to use. In the area I work on, groupby, using categorical data has seen a lot of fixes.
The main reason in releasing pandas 2.0 and not 1.6 is that in major version changes (1 -> 2) is when users expect to have breaking changes. pandas 2.0 is not so significantly different to a 1.6 in terms of features. The main difference is that you really want to make sure that you don't have FutureWarning in your pandas code before upgrading your pandas version.
Awesome. I get scared when I see the red future warning dialog box in jupyter labs. Thanks for everything you guys are doing. Pandas is amazing - I use it most days and always enjoy learning new things. Can't wait to explore 2.0!
Getting rid of your FutureWarnings is a really good idea :) So I applaud you for that. Generally, we wanted to get rid of all the deprecations we introduces since 1.0, so we had to do 2.0 at some point. If your code is free of FutureWarnings then you are good to go. We made some backwards incompatible changes, but not many and they are clearly documented in the release notes. https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html#backwards-incompatible-api-changes
I was personally quite surprised that pandas was an important tool used to obtain the first image of a black hole. I was lucky to meet some of the scientists behind it and learn from them, and their work is much more impressive than what it sounds.
i think one of the hardest things about using pandas is that the core classes have a gazillion methods attached to them, which makes it extremely difficult to navigate the tooling if you're not already intimately familiar with it. I've been using pandas basically since it was created, and I still find myself often needing to reference documentation just to find the method name I need since the output of dir() on any object generally gets truncated.
does any of this resonate? is anyone on your team thinking about ways to improve discoverability of functionality? will there ever be a point at which the team decides there's too much stuff being carried around by too few classes? what are your thoughts on the design philosophy of the tidyverse in juxtaposition to pandas?
Fully agree on this. There are too main things. The first is finding a better API, which is not trivial, and having the functions too divided may not be ideal for some users who prefer `df.whatever()` for everything. Second is that even if we have a better alternative, we may break tens or hundreds of thousands of pandas programs, that won't work after the changes. And we will make millions of users have to relearn the API.
That being said, I'm thinking about a proposal to for example standardize all I/O methods under a `DataFrame.io` namespace (e.g. `df = pandas.DataFrame.io.read_csv(fname)`). More research is needed, and it'll be challenging to reach an agreement with the whole team about this. But maybe 10% of the DataFrame methods you're mentioning would live in a separate and intuitive namespace. There is always a trade-off, and in this case it's clear. Difficult to decide what's best.
Just on this specific example, I think if something is used incredibly often, it should not be put under a namespace like above. .read_whatever is a great example since it is almost always going to be your first call
"readers"/"writers" should return or accept dataframes as I/O types, but should not be methods themselves. There are a lot of "data logistics" methods on dataframes that should be utility functions of the library. Dataframes should only operate on themselves, for analysis or creating/removing/filtering data. A container. A smart container, but just a container.
+1 this is an excellent point I'd never given much thought. I find myself referencing pandas docs more than any other and use it for about 1/4 of the overall code/libs.
I basically live in the pandas docs whenever I use it. I think the library optimizes too much for readability. Whenever I look back on pandas code I've written, the solution is concise and elegant and easy to understand, but it disguises how long it took me to get to that small chunk of code.
I love hearing this! At times I find myself wondering how much our users are utilizing our documentation (especially when compared to some of the great pandas tutorials that are out there). Hearing things like this makes me much more motivated to spend effort there.
Hi there! Long time pandas user -- really appreciate all the work you've done.
I'm only slightly familiar with changes intended in pandas 2.0, namely the switch away from a numpy backend to apache arrow. Historically, the thing I absolutely love about the python numerical stack, is that nearly everything builds off numpy arrays, creating an easily transferable knowledge base between projects.
This is a huge boon compared to other systems where I work (namely R), where there is often more fragmentation in the ecosystem, making interoperability or bespoke analyses much more difficult. Of course, fragmentation in the Python ecosystem has become more common with things like PyTorch tensors, etc.
As an end user, am I going to be losing the numpy < - > pandas interoperability in 2.0? Please feel free to correct any inaccuracies on my end.
Not at all. NumPy is not only staying in pandas 2.0, but it'll still be the default.
That being said, if in the very long term NumPy is eventually dropped, I think exporting from Arrow to NumPy (in our end, not that you'll need to do it) is not only easy, but I think in most cases it can be done without copying (extremely fast, even for huge data). The thing is that NumPy data types are more limited, mostly numeric. If you want to export a string column to NumPy, that's a different story, but there is probably no good reason you want to do that. But for the types that NumPy support well, getting a NumPy array from Arrow backed data won't be a problem. But as said, in pandas 2.0, nothing changed, unless you want it to change and you ask explicitly for pandas types.
There has been some work to make pandas and Polars share data (open a pandas dataframe with Polars, and the other way round). You can read more about it at the end of this post. Not sure if there is any other integration that makes sense, any idea?
Try to spend some time understanding the internals, as you make progress with pandas. Not at the beginning, when you'll have too much to learn just with the basics. But as you become more familiar, it's good to have an idea of what's really happening, in particular when things aren't intuitive. Things like missing values, the infamous copy warning...
6 years working with pandas I still have the docs open every day for simple things. And especially for all those long to wide and wide to long (unstack, stack, pivot etc...) transformations.
Depends on what you're doing, but I'd recommend learning some of the functions for quickly looking at your data. Things like df.head(), df.shape, df.T, etc. From there, learn how to filter your data with df.loc.
Also look into tools like jupyter which make it easy to iterate and visualize data.
Are there any improvements that are coming by way of working with larger datasets/operations without consuming available RAM? I struggle with workarounds when dealing with large data on my 24GB RAM laptop.
Awesome work by the way, Pandas is amazing and we appreciate the work you guys do.
Being able to use Arrow as a backend for your data can save a significant amount of RAM in some cases. Also there is a lot of work related to copy-on-write, that will avoid copying the data when not needed, and will also help reduce the memory needs.
24GB is around 3 billion 64 bits values if I made the numbers right. There is surely some overhead, but with 100 columns you could store around 30 million rows. The main thing wouldn't be only storing, but if you do operations that make a copy of a significant part of that data.
Obviously you may have strings and other things using more than 64 bits per cell, but just to give you an idea of numbers.
I would be interested in this too. I'm running a system with 128 GB of RAM and had quite a lot of difficulty with a 8GB CSV with various permutations of the read_csv() method. I'm sure it is not optimal but would be curious if very large data reads are tested since large amounts of RAM is becoming more common, even on dev workstations, in particular with ML work.
Are you able to change the format of your data on disk? If possible, I would recommend parquet. You'll get smaller file sizes, faster load times, better dtype handling (int vs string), the ability to partition your data sets, and the ability to only load particular columns. Plus peak memory usage should be much lower.
I just started working with pandas two weeks ago, there is so much for me to learn and unpack there so I don’t have a question. Just wanted to give you a shout out for your awesome body of work.
I work with pandas quite a bit for geospatial data analysis, weather data mostly. Because of the higher dimensionality of the data I typically stack the dependent variables into the index as a multi-index [T,X,Y].
Recently I’ve been working with Generic[Enum] types to type annotate the columns inside of a DataFrame.
What kind of support will 2.0 provide for type annotations. One thing I’ve found as a particular annoyance is disconnect between numpy and pandas typing. Where I have to explicitly state the dtype for NDArray[np.int_] and Series[int] and can’t use a TypeVar DType.
I have worked with xarray in the past and quite enjoy it’s built in capabilities for working with weather data, specifically grib2. Unfortunately our pipeline is pretty strict on what dependencies are and are not approved.
There is a typing effort that is led by some core members (unfortunately none of them takes part today). You can check the stubs package out at https://github.com/pandas-dev/pandas-stubs. I am not really familiar with the progress there
I frequently work with pyspark, and although I don't use this feature I know it has support for "pandas udfs" while using arrow behind the scenes.
Now that arrow will be integrated into pandas, do you think we will see improvements in this area? (Performance improvements more features between spark and pandas)
I think it'll take a while, but hopefully we'll eventually see more feature sharing between libraries given we all use Arrow internally. Arrow itself has the concept of kernel, that it's a computation that can be applied to Arrow data. And those can be reused by any library. And the same would apply to user defined functions (udfs). That being said, pyspark is probably using the Java implementation, while pandas is using PyArrow. So, I guess difficult to share many features (I'm not an expert on the JVM, not sure if you could easily call C++ code from a scala program).
From our docs, it appears the keyword on encoding was perhaps at one point used with xlwt (a writer that is no longer maintained) but today is not actually used by pandas. That parameter has been removed in pandas 2.0.
Yes we really like new contributors. There are so many things that need attention, including topics that are well suited for beginners. Tests and Docs specifically come to mind.
Theoretically we can support arbitrary PyArrow types. But we still have some problems regarding nested data in general when doing operations on them. It's a work in progress.
What would you like to do with where? Could you give an use case?
Thanks for the AMA. How do you expect performance to evolve in pandas 2.0 and beyond? I tend to flip between python+pandas and R+data.table. For large datasets (10M-50M rows) I find that group_by/aggregate type operations are an order of magnitude slower in pandas, especially as the number of unique subgroups increases. Also I find memory usage to be significantly higher and well and tend to resort to techniques like downcasting column types after reading to keep memory consumption down, whereas R “just works” for the most part.
Do you view this as a fair comparison? Is performance a first-order concern for you and your team? Hoping to move more from R to Python one of these days but pandas speed is definitely holding me back.
Memory: Copy-on-Write should help reduce the memory footprint through avoiding a bunch of copies. Haven't tested that aspect myself though.
Performance: We are definitely concerned with performance, but not to an extend where we would introduce major breaking changes to get a 10% speed up in some areas. We have to weigh advantages and disadvantages very carefully. That said, we hope that we can get a nice performance boost in some areas where NumPy did not perform that well with our Arrow-backed DataFrames.
I'd recommend trying Copy-on-Write and Arrow backed DataFrames independently and together in 2.0.
What exactly are you doing with groupby that is so slow? This would be interesting to us.
Thanks for the response! I’ll try to generate a representative example of once I get back to my personal laptop and share the performance of R vs Python. I am not doing anything more complicated than aggregating a bunch of columns. I already forego pandas’ more expressive but slower apply() syntax, which I think is the closest in functionality to what R provides.
Thanks a lot for coming here. I am a huge pandas user and I feel excited about this release. I have a question not related to pandas, but maybe you know the answer. In my case I use pandas for EDA and preprocessing before feeding a model from sklearn or tensorflow. Do you know if a dataframe based on arrow will be compatible with sklearn/tensorflow? Or of there are plans to make them compatible?
And what about all numpy methods, do you know if the arrow API is similar to the numpy one? Maybe, thanks to duck typing, many things could still work the same...
The Arrow API is quite different I'd say. One of the bigger differences is, that an Arrow array is not mutable, which is the opposite of NumPy.
We are dispatching to an interface that we can define on a per array basis, so we can get to a point where most of the things work similar/are doing the same.
The size of pandas codebase and features like integrated graphs and stuff made it more difficult to be able to change the backend from numpy to pyarrow? With pyarrow are we going to have any difficulty or limitation working with numpy?
The main issue about our arrow integration is, that we have areas in our code base that explicitly assume that they are working on a NumPy array. This is a bit similar to our non-nanosecond date time resolutions (many functions assumed that the timestamp is given in nanosecond resolution, so changing this is tedious work). We are actively working towards getting rid of these assumptions. That said, we won't break anything in the NumPy world while doing this. The new Arrow functionality shouldn't impact NumPy users in any way.
Thanks for your great work. As a long time pandas user I literally owe half of my career to you. I have two questions:
Reading what Marc wrote about Pandas 2.0 and Arrow, how has the emergence of Polars affected your decisions around Pandas 2.0?
Programming errors are bound to happen. While for small projects, the feedback cycle between writing code and finding errors during runtime is small, this approach doesn't scale very well for large applications, due to their complexity and cost of testing. Are there any plans that allow for more static analysis of Pandas code?
Yes what Marco said regarding typing. I used our stubs and mypy at work and we found a bunch of bugs that we would have missed otherwise. Development there is still ongoing, because it's a big effort :)
Polars did not affect much in our decision making (or anything, I am not aware of anything). Moving more to Arrow is just a natural step for us after we started implementing a StringDtype that is backed by PyArrow a while back. The Arrow community is growing a lot so it made sense for us to integrate better with them
This prevents pandas from being used properly in a WSGI web-server environment due to numpy requiring an interpreter is never loaded twice. But that's exactly what happens in a WSGI Flask Application, so using pandas in a Flask application will break it. Thanks.
As of right now, all of our DataFrames are still backed by NumPy arrays by default. But even if you would use a PyArrow backing, there is still a lot of code where we have to fall back to NumPy, because the Extension Array interface does not support all operations yet.
As long as NumPy is the default backing, there is no real way of avoiding a dependency
As someone (currently) a student who's used open source packages for some time now (mostly Pandas and Django, alongside many many others), I've been looking to contributing to them. I've started to go through the contributing page, as well as reading some of your blogs. Also stumbled across your Google Calendar for meetings - will be there soon!
In general, what have you found is the biggest obstacle (technical or otherwise) preventing aspiring contributors from progressing? What have people done right, and what have people done wrong? Personally, if I was hitting a wall in any way at all, I'd hesitate to ask (whether on slack / otherwise) mostly because the goal is to contribute to this effort - but taking up another member's time to deal with a newcomer might do the opposite. Any thoughts on this?
Setting up the environment takes newcomers quite some time. I have seen this personally when doing sprints. Also finding the correct issue is not easy, but for this I'd suggest searching for "good first issue" and to focus on areas of the code where you are familiar with the interface.
It's totally fine if you have to ask for help, getting started isn't easy when you are not familiar with the processes in open source. That's what our newcomer channel in slack is for.
Really depends on what you are doing. pandas is great if your amount of data isn't too large. It's mature, better tested our API is more stable. Also, we have lots of convenience features build in (this could be a positive or a negative ).
If your data outgrow a certain size, then you are probably better of switching to another tool, there are many out there, Polars, Dask, ...
Right now, I'd be more likely to use pandas in production. Polars is still pretty new, so it does not have the support and integration that pandas has right now. This will probably change over time though.
Do you mean on large data? This is hard for us because of how pandas is build. Dask would be a better alternative if your data don't fit comfortably into memory.
Small and medium sized dataset: Copy-on-Write should help a lot in improving performance if applied correctly (I will write a post explaining the details a bit more soonish that will help with that). Also, using Arrow as a backend get's us a performance boost in operations where Arrow is doing really well (string manipulation for example). Hope this answers your question.
It doesn’t have to be an either or choice. I think they both have a significant place in the python data ecosystem. See some of my precious thoughts on the topic
For beginners and new projects, does it make sense to move to 2.0 right away? If so, what options should be enabled (e.g., nullable_dtypes, dtype_backend, copy_on_write), assuming that the goal is reasonable stability and modernity?
I built and teach a workshop for other academic researchers, and I'm currently doing an update (ahead of the next session in June). Pandas is a big part, though the course ranges from beginner Python language topics to data retrieval/wrangling. The core of this content update is moving from local conda installs to devcontainers/Codespaces, but it would be a good opportunity for me to move to pandas>=2.0.0 if that's a good idea on this timeline.
I'd recommend turning Copy-on-Write on. We expected to make it the default in 3.0 and it basically protects you from doing weird things (ChainedAssignment, mutating multiple objects at once, ...) and should give you a nice performance boost.
I'd definitely advise your students to explore ``dtype_backend`` and ``nullable_dtypes``, depending on the specific use-case it might be a bit early for Arrow (if you are doing a lot of groupby ops for example), but in general at least trying it is a good idea and worthwhile. Really depends on your use case, it's still pretty new but also a nice improvement
What I would recommend your students in every situation is to use the newest version possible if not restricted by something else. Upgrade just get harder if you are already starting on an old version.
Great, thanks! I'll turn that on in my notebooks and see what happens.
Research papers have a natural lifecycle of a few years, typically with a lot of work up front. So, I generally suggest that they start with the newest versions, pin them, and upgrade if needed (usually for a feature or bugfix) while looking out for errors or data changes.
That gives stability/reproducibility within projects (i.e. papers), and upgrades across projects, as newer projects start and older ones are (hopefully) published.
That's not to undercut the value of upgrading projects unnecessarily as a form of procrastination (also known as my plans for some day shortly after 2.0 release).
As an aside, it's really great to see the Arrow stuff coming to fruition in pandas (and over in polars, too). I've been around long enough to remember pandas2/10 things, and the excitement around what the future could look like. A huge congrats to you all for getting a big part of that over the line.
Yes I definitely agree with pinning your versions.
What I did at work: We pinned our environments that we were using and did an upgrade every 6-8 months. We tested on an unpinned version of our environment in parallel to see issues quicker. This made upgrading easy and pretty efficient. My comment was more meant towards using 1.3 now and the wondering in a year while upgrading takes forever.
If you only need your stuff for a year or something similar, then this is not that big of a problem.
One thing I've never really grasped is why multi-indexed results are returned rather than flattening. I have never had a circumstance where I've used that sort of result directly, and always have to figure out how to get rid of the "outer" index. What was the reasons for this multi-index, and not just a rectangular dataframe?
My apologies, its been a couple months, but I believe group_by operations were often the culprit. I never felt like I got the group_by -> aggregate -> group_by -> aggregate workflow figured out, which I use constantly to summarize replicate data in an experiment and then summarize at the treatment level. This was probably a me problem though!
It may stem from me having started with R-tidyverse, which as you likely know shares much of the same namespace, but often operates just differently enough to make transitions between tidyverse and pandas just a bit of a headache.
As a total noob to pandas, what makes it substantively different than the python standard library? For instance I can create lists of lists or dicts of lists. Drop/add items etc. I can combine lists of lists too.
I'm not trying to be a smart*ss, I'm genuinely asking: what's the differentiator? Pandas is a bit heavy. If I can write some classes that kind of handle those operations and a few more with syntatic sugar, why install pandas?
Assuming you care at least a little bit about performance: Pandas and numpy are vectorized and the actual operations happen mostly in C and cpython which makes it way faster than anything you could do in only python. If you would implement this in c, then you’d build a new numpy 😀 This is a really short answer but hope that you get the gist of the difference
I'm unsure what's the support for decimal in pandas right now. One thing is to be able to load Arrow columns in pandas, and the other is what operations for that data type are implemented. In any case, if not all what you need is in pandas 2.0, it'll come eventually. Particularly if you open issues and PRs in our issue tracker.
That being said, you can do like the UK stock market, just have all the amount in cents, and you can do it with integers. ;)
In general, are the plans to have the rolling API more closely align with the rest of the pandas API? In particular, are there any plans to have df.rolling.groupby() return similarly indexed results as a normal df.groupby()?
E.g., with the latter you have the wonderful .transform() method to add a column to the df. When working with the rolling window, you always get a MultiIndexed dataframe that is much harder to align to the index of the original df.
Perhaps (hopefully?) there are better ways, but I currently use a combination of extracting a single column as Series, using groupby(as_index=False) and finally a call to set_axis(df.index) to get the desired result to align with my original dataframe.
I work almost daily with Pandas so I definitely want to give me thanks and appreciation for this excellent tool.
Any plans for built-in parallelization in Pandas? I know there are many modules attempting to implement this with varying success, like pandarallel, dask or swifter. However I had difficulty getting any of these to work in an existing application without major refactoring.
In our case, we have a high level application class or processor that ingests many dataframes which sit in memory as properties to the processor instance. This processor does various processing to different dataframes in conjunction with eachother, like iterrows or applys on one dataframe while checking other dataframes which are all unique attributes of the same object running in memory concurrently.
However when the processor class actually runs, ultimately everything is stuck in a single core but I would say most systems have at least 6 or more cores now, even cheap laptops. Having a model or two to apply parallelization using concurrent.futures based on threads or processes seems like it would make a lot of sense. I think threads would likely work well if implemented intelligently, but I'm sure I am oversimplifying.
Supporting multithreading would be really really cool, but this requires a lot of effort. There is some considerations in that area but nothing imminent unfortunately.
I think Arrow should help make this easier. It'll depend on each particular case, but read_csv is already parallel when selecting the pyarrow engine. Parallel computing is never easy, but I think we should be able to slowly parallelize more operations.
Historically, pandas has relied on other libraries in the ecosystem to support parallelization such as https://www.dask.org/ which uses pandas under the hood. One thing to also keep in mind is that certain NumPy operations (which pandas uses) may be parallel depending on how your BLAS (Basic Linear Algebra Subprograms) are setup. In general, you want to avoid having multiple levels of parallelism which can actually hurt performance.
I would also recommend avoiding iterrows or applys if you can vectorize your operations - you will see very significant performance benefits. But depending on what you're doing, that may not be possible.
This doesn't really answer the question, but whenever you see two different implementations doing the same or similar things, you can carefully compare each step in the implementation. This very often reveals hard to find bugs in one or both of the implementations.
I can't recall a time I found a bug and made it into a feature.
What if I have a problem where I am just ITCHING to iterate over an entire dataframe row by row via itertuples for simplicities sake, and map(lambda) isn’t feasible? What would you recommend?
Iterating a dataframe is slow. If speed is important, you should try to build your pandas code in a way that you never implement loops, but delegate to pandas the operations, so they happen fast in C, and not via the Python interpreter.
If you iterate the data, then you're just in regular Python, with a Python tuple object, and you can write any code that is valid Python. Not sure in what case map() wouldn't be an option, but you can always replace a map by a loop (or a comprehension) when you're in Python.
I already did this, and realize it's probably an abomination, but:
How would you go about enforcing columns to have certain types? And when a column has a list in it, that each entry of the list is a certain type?
I accomplished this by making a new class inheriting from DataFrame as well as pydantic's BaseClass and used those as validated rows to then shove into a DataFrame. Messy but it works! Maybe you have a better idea.
With the integration of Arrow will pandas support Structs from Bigquery better? Like having the types and understand lists inside columns, better visualization of them... I remember the last time I had to deal with them, couldn't make pandas understand what was inside the column with an .astype(...), I checked the code of bigquery and they were getting the data into an arrow table, and creating the schema to use it while exporting to a pandas dataframe
I have kind of a broad question. I work as an ETL engineer, and the vast majority of my current work is with CSV files (sometimes xlsx). In my experience, a lot of people in the ETL space default to pandas because of the simple "read_csv/xlsx → to_sql" pipeline and the speed of vectorized operations. I also suspect there's a "popular because it's popular" thing happening to some degree. I've developed (and cultivated) a bit of a reputation as a pandas hater for a few reasons:
We do basically zero numerical analysis or aggregation during ETL
Every validation happens within a single record (i.e. no record has any bearing on any other record), so we end up using iterrows a lot
Lack of support for strings and None (dealt with in 2.0)
Ultimately, I feel that pandas is a hammer that makes everything look like a nail specifically for ETL. I have used pandas for data analysis and really appreciate its usefulness and how easy it is to teach to others (even if they don't know Python). But when we're talking ETL, it just isn't the right tool...right?
Anyway, that's barely a question, but if you have time I'd love to hear your thoughts on pandas usage in non-data analysis spaces. Also I want to qualify this by saying I am far from an expert and I'm really just looking to learn! Thanks for your time.
It's hard to say without knowing what validation you're doing, but my experience is that even very complex operations can be vectorized. If you are able to do this, whether with pandas or something else, you'll experience significant performance benefits.
You mention not using aggregation, but pandas can also be very efficient at reshaping data - though I don't know if you might have a use for that in your ETL.
You make some very good points. pandas was designed as both an ETL tool, and a data analysis tool. I wrote an article about it long time ago. But summarizing, it won't ever be able to master both. And feels like many decisions were made more for the data analysis tool.
For an ETL tool I'd expect things to never fail silently, type conversions never happen automatically, schemas have to be provided and not inferred...
For your use case, pandas may be an overkill. DuckDB could be a nicer option if your validations can be vectorized. But if you're going to iterate row by row, I'd personally just write your ETL in Rust (may be trickier for the xlsx, not sure if there are good libraries for it).
To answer your question though, there is quite a lot. If you are using pandas 1.5.3 and seeing an FutureWarnings, that is something that will change or break when you switch to 2.0.
There is not much impact in pandas 2.0 regarding numba/cython.
We fix small inconsistencies to the pandas API, but we avoid changing it too much, since we consider that the cost in users having to migrate code and relearning things is too much.
48
u/jabies Mar 01 '23
How does the Pandas project address the open source funding problem? Do you want pandas devs in their dayjobs to nudge management to sponsor somehow?