r/ApacheWayang • • Jan 03 '23

Join Apache Wayang and change the world of data

2 Upvotes

You know data mesh? And you are looking for a great place to build the future of decentralized ML / AI / data processing behind boundaries? Than look into Apache Wayang and join the developer community: https://wayang.apache.org/community/

Apache Wayang is the next stage of big data processing, a successor to Apache Spark. It's also called the API for BigData, and used by the top software companies around the world. 

https://www.polywork.com/opportunities/fTwktxcp


r/ApacheWayang • • Mar 22 '22

Research papers online

1 Upvotes

Hey community, we recently released all research papers conducted to build and release Apache Wayang via our startup:

https://www.databloom.ai/science

Enjoy ;)


r/ApacheWayang • • Mar 17 '22

What is Apache Wayang?

4 Upvotes

What is Apache Wayang? We explain it with - the API for BigData. Never care about the processing framework anymore, code against the Wayang API. Easy.

Apache Wayang has a three-layer data processing abstraction that sits between user applications and data processing platforms, such as Hadoop and Spark. The figure below depicts the Apache Wayang architecture: (i) an application layer that models all application-specific logic; (ii) a core layer that provides the intermediate representation between applications and processing platforms; and (iii) a platform layer that embraces the underlying processing platforms. Overall, the input of an application layer comprises the logical operators provided by users (or generated by a declarative query parser) and the output is a physical plan (WayangPlan). The WayangPlan is then passed to the core layer where cross-platform optimizations take place to produce an execution plan (ExecutionPlan).

More informations: https://wayang.apache.org/about/


r/ApacheWayang • • Jan 18 '23

How can I build a FL stack using Apache Groovy and Apache Wayang?

1 Upvotes

Building a Federated Learning (FL) stack using Groovy and Apache Wayang can be a complex process, but it is definitely possible. Here are some general steps that you can follow:

  1. Install Groovy: To use Groovy, you'll need to first install it. You can find the installation instructions on the Groovy website (https://groovy-lang.org/)

  2. Install Apache Wayang: To use Apache Wayang, you'll need to first install it. You can find the installation instructions on the Apache Wayang website (https://wayang.apache.org/).

  3. Prepare your data: To train a model using FL, you'll need to have data on each device or client you want to use for training. You'll need to format your data so that it can be used with Apache Wayang.

  4. Define the model architecture: You'll need to define the architecture of the model you want to train. Apache Wayang provides a set of pre-built model architectures that you can use or you can define your own.

  5. Define the FL training configuration: You'll need to define the FL training configuration, such as the number of clients, the number of rounds, the learning rate, etc.

  6. Start the training: Once you have prepared your data, defined the model architecture, and set the FL training configuration, you can start the training process by using the Apache Wayang API, which is in Java. You can use groovy to call the Java API

  7. Monitor the training: You can monitor the training process using the Apache Wayang API, to check the progress, accuracy, and other metrics.

  8. Deploy the model: Once the training is completed, you can deploy the model on the devices or clients.

It's important to note that, as with any software development project, it's important to have a solid understanding of the problem you're trying to solve, the data you're working with, and the specific requirements of your use case. Also, it's important to have a solid knowledge of the Groovy language and its features, as well as the Java language, as Apache Wayang is a Java based framework. It may be helpful to look at existing examples provided by Apache Wayang and the documentation provided on the project's website to get a better understanding of how to use the API and implement FL with Apache Wayang using Groovy.


r/ApacheWayang • • Jun 19 '22

Find the best single malt with Apache Wayang:

Thumbnail blogs.apache.org
1 Upvotes

r/ApacheWayang • • May 10 '23

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • May 25 '22

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Jun 07 '23

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Jul 05 '23

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Nov 30 '22

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Oct 05 '22

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Mar 24 '23

Join the Apache Wayang Project - An Open-Source Federated Data Framework

1 Upvotes

Are you interested in distributed machine learning and looking for an open-source project to contribute to? Check out Apache Wayang! It’s a Federated Data framework that provides a platform for distributed data processing with ease of use and flexibility. It supports multiple FL scenarios and includes tools and components for building FL systems, as well as support for various communication protocols and data formats. Come join our community on Github (https://github.com/apache/incubator-wayang) and help us build something amazing! Learn more via wayang.apache.org


r/ApacheWayang • • Apr 05 '23

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Jun 21 '22

Tutorial 2: Developing with Blossom Development Environment and Apache Wayang

1 Upvotes

r/ApacheWayang • • Sep 13 '23

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Mar 30 '22

The Missing Piece in ML-based Query Optimization

1 Upvotes

Machine Learning (ML) has not only become omnipresent in our everyday lives (with self-driving cars, digital personal assistants, chatbots etc.) but has also started spreading to our core technological systems, such as databases and operating systems. In the area of databases, there is a large amount of works aiming at optimizing data management components, from index building, knob tuning to query optimization. Just in query optimization, ML is used in the place of many optimizer components, such as cardinality estimation, cost model, and join enumeration. In this blog post, we focus on the case of using an ML in the place of a cost model and go from the traditional cost-based query optimization to the newly proposed ML-based query optimization.

Blogpost via databloom.ai:
https://engineering.databloom.ai/2022/03/the-missing-piece-in-ml-based-query.html


r/ApacheWayang • • Nov 02 '22

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Nov 16 '22

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Apr 07 '22

Python interface in beta Spoiler

3 Upvotes

The dev team is working on the last regression tests for our long anticipated python API. We expect GA May ‘22, at the latest. With python enabled, Wayang now distributes data workloads also to Tensorflow federated, and more important, we enable the whole data science community to work with their most favorite tools. Imagine Huggingface with Spark!


r/ApacheWayang • • Jun 30 '22

Towards a Learning-Based Query Optimizer

Thumbnail news.ycombinator.com
1 Upvotes

r/ApacheWayang • • Oct 04 '23

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Feb 08 '23

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!


r/ApacheWayang • • Jul 12 '22

New readme up, we also build an easy installer =>

Thumbnail
github.com
1 Upvotes

r/ApacheWayang • • Jun 08 '22

New Members Intro

1 Upvotes

If you’re new to the community, introduce yourself!