r/databricks • • 7d ago

Discussion Using the Apache DataFusion Comet accelerator to save cost

I wonder if anybody has utilized the comet accelerator to speed up things and therefore save cost? Given the photon tax this seems like obvious choice but I'm not sure if the project is already mature enough.

6 Upvotes

4 comments sorted by

2

u/Youssef_Mrini databricks 5d ago

Why do you think Photon is expensive ? did you run your job before and after photon and compared the costs ?

1

u/daddy_stool 3d ago

Yes many people think Photon is too expensive.
Especially if you compare with newer engines like LakeSail.

1

u/SpecificTutor 7d ago

we run comet over the vanilla spark k8s operators. its been fairly stable and a definite improvement over dbr spark.

1

u/Master-Performer-642 22h ago

Is most of your Photon spend on jobs compute or on SQL warehouses? The answer changes which levers you have