r/databricks • u/22Maxx • 7d ago
Discussion Using the Apache DataFusion Comet accelerator to save cost
I wonder if anybody has utilized the comet accelerator to speed up things and therefore save cost? Given the photon tax this seems like obvious choice but I'm not sure if the project is already mature enough.
6
Upvotes
1
u/SpecificTutor 7d ago
we run comet over the vanilla spark k8s operators. its been fairly stable and a definite improvement over dbr spark.
1
u/Master-Performer-642 22h ago
Is most of your Photon spend on jobs compute or on SQL warehouses? The answer changes which levers you have
2
u/Youssef_Mrini databricks 5d ago
Why do you think Photon is expensive ? did you run your job before and after photon and compared the costs ?