r/ApacheWayang • • Apr 12 '22

Python API done, python as platform mostly done

8 Upvotes

We work since nearly a half year on full python platform support. Great news - we solved the JVM - python UDF problem! In our first tests we could directly interact with Tensorflow and Huggingface - Wayang is the API for big data analytics and AI :)

https://github.com/apache/incubator-wayang/commit/f738e66fd4db66b08c2c0e67f35ca3101c9a3bf5


r/ApacheWayang • • Mar 17 '22

What is Apache Wayang?

4 Upvotes

What is Apache Wayang? We explain it with - the API for BigData. Never care about the processing framework anymore, code against the Wayang API. Easy.

Apache Wayang has a three-layer data processing abstraction that sits between user applications and data processing platforms, such as Hadoop and Spark. The figure below depicts the Apache Wayang architecture: (i) an application layer that models all application-specific logic; (ii) a core layer that provides the intermediate representation between applications and processing platforms; and (iii) a platform layer that embraces the underlying processing platforms. Overall, the input of an application layer comprises the logical operators provided by users (or generated by a declarative query parser) and the output is a physical plan (WayangPlan). The WayangPlan is then passed to the core layer where cross-platform optimizations take place to produce an execution plan (ExecutionPlan).

More informations: https://wayang.apache.org/about/


r/ApacheWayang • • Apr 07 '22

Python interface in beta Spoiler

3 Upvotes

The dev team is working on the last regression tests for our long anticipated python API. We expect GA May ‘22, at the latest. With python enabled, Wayang now distributes data workloads also to Tensorflow federated, and more important, we enable the whole data science community to work with their most favorite tools. Imagine Huggingface with Spark!


r/ApacheWayang • • Mar 27 '22

EarthCube, based on Apache Wayang, will be introduced at VLDB 2022!

3 Upvotes

r/ApacheWayang • • Mar 17 '22

The wayang team is working on SQL integration

3 Upvotes

We want to share some development insights today: We are working on a native SQL layer for Apache Wayang.

Extending Wayang with Apache Calcite (calcite.apache.org) enables our users, customers and community to use native SQL in their model; it's not a simple stretch to use sql queries, we enable cross-platform processing on data warehouses like Snowflake, Starburst, Microsoft, Redshift and much more. Saying this, we extend federated learning across all domains our users master - from Tensorflow to SQL, without the need to re-engineer the code.

Watch for more news in the coming weeks!

https://github.com/apache/incubator-wayang/commit/55e972c4966d4e060e545d41b609217b2acf2d36?fbclid=IwAR1Qyj1O-kNiCMSpYKSHh6MH58JwBo9w3bfkh9MOgGuI81JBFDyrJvZzSbE


r/ApacheWayang • • Dec 05 '23

Support different jdbc platforms and multiple instances of same DBMS

2 Upvotes

We merged PR380, enables multi-spark context per call and jdbc data queries to different systems in one wayangPlan(). You can now query multiple spark instances and multiple different databases at once.

This enables federated learning on industrial level, across databases and data lakes. Model training at a fraction of the current costs coming soon!

https://github.com/apache/incubator-wayang/pull/381


r/ApacheWayang • • Aug 30 '23

New Members Intro

2 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Apr 16 '23

Apache Calcite support added to Wayang!

2 Upvotes

Initial version of the SQL API by kbeedkar · Pull Request #284 · apache/incubator-wayang (github.com)

Exciting news! Apache Calcite support has been added to Wayang, with more operator and data mesh functions on the way. This makes Wayang the first engine capable of fully operating across multiple, fundamentally different data processing frameworks.


r/ApacheWayang • • Jan 03 '23

Join Apache Wayang and change the world of data

2 Upvotes

You know data mesh? And you are looking for a great place to build the future of decentralized ML / AI / data processing behind boundaries? Than look into Apache Wayang and join the developer community: https://wayang.apache.org/community/

Apache Wayang is the next stage of big data processing, a successor to Apache Spark. It's also called the API for BigData, and used by the top software companies around the world. 

https://www.polywork.com/opportunities/fTwktxcp


r/ApacheWayang • • Jun 26 '22

Kandi reviewed Apache Wayang

2 Upvotes

https://kandi.openweaver.com/java/apache/incubator-wayang

"In contrast to traditional data processing systems that provide one dedicated execution engine, Apache Wayang (incubating) is a cross-platform data processing system: Users can specify any data processing application using one of Wayang's APIs and then Wayang will choose the data processing platform(s), e.g., Postgres or Apache Spark, that best fits the application. Finally, Wayang will perform the execution, thereby hiding the different platform-specific APIs and coordinating inter-platform communication."


r/ApacheWayang • • Apr 20 '22

New Members Intro

2 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Apr 18 '22

Over 2.5k active monthly users using Apache Wayang!

Post image
2 Upvotes

r/ApacheWayang • • Mar 22 '22

Poisoning attacks in Federated Learning

2 Upvotes

Federated learning is a double-edged sword in that it is designed to ensure data privacy, yet unfortunately, it opens a door for adversaries to exploit the system easily. One of the popular attack vectors is a poisoning attack. Read the blogpost to get more insights:

https://engineering.databloom.ai/2022/02/poisoning-attacks-in-federated-learning.html


r/ApacheWayang • • Mar 17 '22

Welcome to our Apache Wayang community

2 Upvotes

We are the team around Apache Wayang (wayang.apache.org). Here we post updates, discuss with users and community members and provide help. Help us to grow the community!


r/ApacheWayang • • Mar 17 '22

r/ApacheWayang Lounge

2 Upvotes

A place for members of r/ApacheWayang to chat with each other


r/ApacheWayang • • Apr 01 '26

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Mar 18 '26

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Mar 11 '26

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Mar 04 '26

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Feb 25 '26

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Feb 11 '26

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Feb 04 '26

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Jan 28 '26

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Jan 21 '26

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!