r/pythonhelp • • Sep 30 '22

How do i parallelize this simple for loop?

I want to precompute a (symmetric) distance matrix to use it later for clustering with dbscan.

However already for a tiny 51x51 test matrix takes ~10min to calculate with the actual datasets i want to cluster being on the order of >2000 points which would take upwards of 10 days to compute.

Due to that i want to parallelize this if possible.

Currently the matrix is being build like this:

        precomputed = np.zeros((len(train_features), len(train_features)))
        for i in range(len(train_features)):
            logging.info(i)
            for j in range(i + 1, len(train_features)):
                precomputed[i][j] = self.trajectory_distance(
                    train_features[i], train_features[j], "geodesic"
                )
                precomputed[j][i] = precomputed[i][j]

I feel like this should be easily be parallelized using multiprocessing.pool but i seem to have a mental block right now.

3 Upvotes

0 comments sorted by