r/learnmachinelearning • u/sarcasmasaservice • Feb 08 '24
Help scikit-learn LogisticRegression inconsistent results
I am taking datacamp's Dimensionality Reduction in Python course and am running into an issue I cannot figure out. I'm hopeful someone here can point me in the right direction.
While working through Chapter 3 Feature Selection II - Selecting for Model Accuracy of the course I find I'm unable to fully replicate the results that datacamp is getting on my local machine and want to understand why.
I have created a GitHub repo with a MWE in the form of a Jupyter notebook or a Python script for anyone who is willing to look at it.
To describe my problem, datacamp and I are getting different results. datacamp consistently gets:
{'pregnant': 5, 'glucose': 1, 'diastolic': 6, 'triceps': 3, 'insulin': 4, 'bmi': 1, 'family': 2, 'age': 1}
Index(['glucose', 'bmi', 'age'], dtype='object')
80.6% accuracy on test set.While my results vary but almost always include the 'pregnant' feature unless I drop it from the dataset.
According to my experiments, datacamp and I are producing identical correlation matrices and our heatmaps are, not surprisingly, identical as well.
Interestingly, if I don't increase the max_iter parameter I would get the following after my results:
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(The value I needed to set for max_iter was not constant but I never saw the error with a value >= 200.
My first thought was that perhaps the default solver has changed was different.
On datacamp:
In [16]: print(LogisticRegression().solver)
lbfgsand on my machine:
>>> print(LogisticRegression().solver)
lbfgsI also checked the version of scikit-learn.
datacamp's version:
In [17]: import sklearn
In [18]: print('sklearn: {}'. format(sklearn. __version__))
sklearn: 1.0and my version:
>>> import sklearn
>>> print('sklearn: {}'. format(sklearn. __version__))
sklearn: 1.3.2My next thought was to try installing scikit-learn v1.0 on my machine to see if I can reproduce the site's results. This, however, turned out to be more involved than I expected due to dependency issues. Instead, I built a separate env with numpy v1.19.5, pandas v1.3.4, scikit-learn v1.0, and Python v3.9.7 to mirror the site's environment. The result is the repo I mentioned above.
I would appreciate *any* insight into why I am seeing different results than datacamp, and why my results will vary from run to run. I'm new at this but really want to understand.
Thanks in advance.
3
Feb 08 '24
[removed] — view removed comment
1
u/sarcasmasaservice Feb 08 '24
Thanks for your reply. Setting a value for
random_statein either my call totrain_test_split()and/orLogisticREgression()I am able to get reproducible results, however they still do not match those of datacamp. Additionally, my results always include'pregnant', a weaker predictor than'glucose'which never seems to make it through the RFE. Any guidance on why this might be?4
u/orz-_-orz Feb 08 '24
however they still do not match those of datacamp.
Is your seed the same as the one in the data camp?
2
u/sarcasmasaservice Feb 08 '24
In this particular exercise datacamp does not state what value they use for
random_state. In other exercises in the same chapter they set it to 0 which is the value I have been using as well since you suggested it. Is there a way to retrieve the value after the model has been initialized? If so, my googling hasn't found it.
2
u/FlivverKing Feb 09 '24
Drop the RFE in your code and run a logistic regression with the same features data camp used. If your coefficients look the same, then the problem is the RFE. If they look substantially different, the issue is the data split. In most real world settings where you’d use a logistic regression over an alternative model, you wouldn’t want to use RFE anyway.