r/pythonhelp • u/JanEric1 • Sep 30 '22
How do i parallelize this simple for loop?
I want to precompute a (symmetric) distance matrix to use it later for clustering with dbscan.
However already for a tiny 51x51 test matrix takes ~10min to calculate with the actual datasets i want to cluster being on the order of >2000 points which would take upwards of 10 days to compute.
Due to that i want to parallelize this if possible.
Currently the matrix is being build like this:
precomputed = np.zeros((len(train_features), len(train_features)))
for i in range(len(train_features)):
logging.info(i)
for j in range(i + 1, len(train_features)):
precomputed[i][j] = self.trajectory_distance(
train_features[i], train_features[j], "geodesic"
)
precomputed[j][i] = precomputed[i][j]I feel like this should be easily be parallelized using multiprocessing.pool but i seem to have a mental block right now.
3
Upvotes