r/awslambda • u/ayuuuss • May 10 '26
Unable to load .pkl ML model for AWS Lambda (dependency/version issues) – tried EC2 also
Hi, I’m trying to deploy a machine learning model on AWS Lambda.
I have:
a .pkl file (saved using joblib)
a lambda_function.py file to load and run predictions
My goal is to deploy this on Lambda, but I was getting dependency issues, so I tried setting it up on an EC2 instance first to debug.
However, I’m facing multiple errors while loading the model, and I don’t have access to the original environment or requirements.txt (my friend trained the model and hasn’t shared it yet).
Errors I’ve encountered:
ModuleNotFoundError: No module named '_loss'
ModuleNotFoundError: No module named 'numpy._core'
ValueError: MT19937 is not a known BitGenerator module
What I’ve tried:
Creating virtual environment on EC2 (Ubuntu)
Installing different versions of numpy, scipy, scikit-learn, joblib, xgboost
Matching sklearn version (1.7.2 from warning)
Re-downloading the .pkl file
Trying Docker build for Lambda image
Still not working.
Current setup:
AWS Lambda (target)
EC2 Ubuntu instance (for testing)
Python 3.10
joblib for loading model
Code:
import joblib
data = joblib.load("clinical_trial_pipeline_v1.pkl")
model1 = data['model1'] scaler1 = data['scaler1']
X = [[1,2,3,4]] X_scaled = scaler1.transform(X) prediction = model1.predict(X_scaled)
print(prediction)
My questions:
Is it possible to recover or infer the correct environment from a .pkl file?
Is this likely due to version mismatch between numpy/sklearn?
What’s the best way to make this work for AWS Lambda without original requirements.txt?
Any help would be really appreciated I’ve been stuck for 2 days trying to fix this.
2
u/13ass13ass May 11 '26
I would attempt to debug the environment with your local workstation first if you can.
Try import pickletools with open("clinical_trial_pipeline_v1.pkl", "rb") as f: pickletools.dis(f)
And see if you can create a requirements txt from your findings there.
Then bc of c lib dependencies I would stick with ec2 or lambda containers and skip the zip file approach.