r/datawarehouse • • Aug 08 '23

Virtual Data Builds: A data warehouse environment for every Git commit

Thumbnail
y42.com
15 Upvotes

r/datawarehouse • • Feb 12 '25

How Data Warehousing and BI Systems Are Evolving

9 Upvotes

Must read: Our comprehensive breakdown of how modern businesses are leveraging data warehouse and BI systems.

Some interesting findings:

  • The primary driver for data analytics implementation is process improvement and cost efficiency (60% of companies)

  • There's a significant shift towards cloud-based solutions

  • The integration between data warehouses and BI tools is becoming more seamless

We covered everything from basic architecture to predictive analytics and cloud evolution. Would love to hear your thoughts - what trends are you seeing in your organizations? Any challenges with integration between data warehouses and BI tools?

Full blog: https://data-sleek.com/blog/leveraging-data-warehouse-and-business-intelligence-systems/


r/datawarehouse • • Jul 03 '20

DWH Warehouse in practice - design question.

9 Upvotes

So I'm looking at moving our in-house BI SQL Server (2019) to more of a Data Warehouse. I've been doing my research and completely get the whole Dimensions/Facts and snowflake/star schema in principle.

However, I have a gap between theory and practice - What do I do with my reporting tables that are a combination of facts and dims? Do I create them as Views or SPs, or do I create them as a sort of 3rd type of table?

For a bit on the context, we've created tables that record historical performance for KPIs and other types of aggregated data such as the number of activities by customers. What do I do with these in a Data Warehouse world?


r/datawarehouse • • Nov 15 '24

Recommendations for some ETL Tools?

8 Upvotes

Hey folks! I’m working on setting up a data warehouse and looking for some ETL tools to help with connecting and automating the process. I’ve heard of tools like Hevo, but I’m curious if anyone has used it or other easy-to-use tools that can handle data from different sources. Any recommendations for something that’s simple to set up and scales well? Would love to hear your thoughts!


r/datawarehouse • • Jan 02 '24

Data Testing Cheat Sheet: 12 Essential Rules

8 Upvotes
  1. Source vs Target Data Reconciliation: Ensure correct loading of customer data from source to target. Verify row count, data match, and correct filtering.

  2. ETL Transformation Test: Validate the accuracy of data transformation in the ETL process. Examples include matching transaction quantities and amounts.

  3. Source Data Validation: Validate the validity of data in the source file. Check for conditions like NULL names and correct date formats.

  4. Business Validation Rule: Validate data against business rules independently of ETL processes. Example: Audit Net Amount - Gross Amount - (Commissions + taxes + fees).

  5. Business Reconciliation Rule: Ensure consistency and reconciliation between two business areas. Example: Check for shipments without corresponding orders.

  6. Referential Integrity Reconciliation: Audit the reconciliation between factual and reference data. Example: Monitor referential integrity within or between databases.

  7. Data Migration Reconciliation: Reconcile data between old and new systems during migration. Verify twice: after initialization and post-triggering the same process.

  8. Physical Schema Reconciliation: Ensure the physical schema consistency between systems. Useful during releases to sync QA & production environments.

  9. Cross Source Data Reconciliation: Audit if data between different source systems is within accepted tolerance. Example: Check if ratings for the same product align within tolerance.

  10. BI Report Validation: Validate correctness of data on BI dashboards based on rules. Example: Ensure sales amount is not zero on the sales BI report.

  11. BI Report Reconciliation: Reconcile data between BI reports and databases or files. Example: Compare total products by category between report and source database.

  12. BI Report Cross-Environment Reconciliation: Audit if BI reports in different environments match. Example: Compare BI reports in UAT and production environments.


r/datawarehouse • • Oct 31 '23

We’ve made Data Quality an engineer’s problem. It’s actually a tooling issue

Thumbnail
y42.com
8 Upvotes

r/datawarehouse • • Dec 31 '22

What is Business Intelligence - Father of Data Warehousing Ralph Kimball

Thumbnail
youtu.be
8 Upvotes

r/datawarehouse • • Mar 16 '19

Consensus on Agile Data Warehousing?

7 Upvotes

I am wondering if there is an industry consensus around how to build a data warehouse in an Agile environment. The Kimball methodology requires a great deal of certainty in the beginning of a warehouse project (through the Enterprise Bus Matrix) and these requirements will change when the business sees the first iteration. Changes will cause the warehouse to be altered, then rebuilt; an expensive operation.

How are practitioners successfully versioning, iterating, and frequently deploying their data warehouse builds to keep up with the changing requirements of the business? I have seen interesting perspectives on the Data Vault modeling methodology but a lot of the websites describing it look old and cheap. Would love some perspective.


r/datawarehouse • • Oct 02 '24

The Complete Guide for Data Warehouse Migrations

Thumbnail
selectstar.com
6 Upvotes

r/datawarehouse • • Apr 15 '23

Business Intelligence 101: From Data to Insights - Part 1

Thumbnail
datafriends.co
6 Upvotes

r/datawarehouse • • Apr 21 '18

Please assist in clarifying Data Warehouse Architecture process

6 Upvotes
  1. Take data from Operational systems.

  2. Integrate Data from Multiple sources.

  3. Standardize data and remove inconsistencies.

  4. Store data in format suitable for easy access.

Operational system ---> (ETL ---> Data Warehouse ---> Olap)---> Business Users

Where does Data mining come in? Is Data Mining during the Olap process?

monitor Integration to Data mining to Olap seem to be a filter but so are Rolap server and Molap server. I am confused. Please assist


r/datawarehouse • • Feb 05 '25

What Data Warehouse & ETL Stack Would You Use for a 600-Employee Company?

5 Upvotes

Hey everyone,

We’re a small company (~600 employees) with a 300GB data warehouse and a small data team (2-3 ETL developers, 2-3 BI/reporting developers). Our current stack:

  • Warehouse: IBM Netezza Cloud

  • ETL/ELT: IBM DataStage (mostly SQL-driven ELT)

  • Reporting & Analytics: IBM Cognos (keeping this) & IBM Planning Analytics

  • Data Ingestion: CSVs, Excel, DB2, web sources (GoAnywhere for web data), MSSQL & Salesforce as targets

What We’re Looking to Improve

  • More flexible ETL/ELT orchestration with better automation & failure handling (currently requires external scripting).

  • Scalable, cost-effective data warehousing that supports our SQL-heavy workflows.

  • Better scheduling & data ingestion tools for handling structured/unstructured sources efficiently.

  • Improved governance, version control, and lineage tracking.

  • Foundation for machine learning, starting with customer attrition modeling.

What Would You Use?

If you were designing a modern data stack for a company our size, what tools would you choose for:

  1. Data warehousing

  2. ETL/ELT orchestration

  3. Scheduling & automation

  4. Data ingestion & integration

  5. Governance & version control

  6. ML readiness

We’re open to any ideas—cloud, hybrid, or on-prem—just looking to see what’s working for others. Thanks!


r/datawarehouse • • Jan 16 '24

Future of Big Data Systems by Spark creator Matei Zaharia

Thumbnail
youtu.be
5 Upvotes

r/datawarehouse • • Jan 04 '24

Which ETL-tool do you use at work?

5 Upvotes

Question in the title. I am honestly interested about what other tools people use for ETL processes within their data-warehouse environments. What are the upsides? Downsides? Would you recommend it?

Let me start: Use: Pentaho, low code visual ETL tool Upsides: relatively easy to pick up for non programmers,free, multithreaded Downsides: clunky, javascript based, little documentation online


r/datawarehouse • • Oct 09 '23

Best practices for working with dbt and Snowflake - A practitioner’s guide

Thumbnail
y42.com
5 Upvotes

r/datawarehouse • • Oct 27 '22

Is a Microsoft Tabular Cube completely outdated?

5 Upvotes

For an On-Premise data warehouse, with Windows servers and SQL licensing already provisioned, is a tabular model still viable? What is a better approach?

This is a retail business with a small data foot print (250gb for now, might add 20gb per year).

Primary purpose is to consolidate data with defined relationships and common metrics. Data consumption will primarily be through Excel, but some scheduled SSRS reports will go out daily to operators.

Going cloud might have sone benefits but it’s difficult to justify when local infrastructure and licenses are paid for.


r/datawarehouse • • Oct 14 '22

Data warehouse from scratch

5 Upvotes

Hello guys!

I need to make a simple data warehouse. There is a stored procedure running on the main DB, it is huge and it has at least 15 tables in it.

I need to move somehow the resultset of this stored procedure to another DB, with simpler not normalized table...

What is the best way to do it? How to track data changes? How to set up the sql agent to run the process every night... Etc etc

There are so many unknowns for me here and I know this is a broad question, so please forgive me. I need some sort of guidance, some set of instructions on what or how to accomplish this. I am willing to pay for consultancy. Thanks


r/datawarehouse • • Apr 21 '25

Begginer's questions - Data duplication through DW stages

4 Upvotes

Hello everyone, I'm starting my studies on data warehouse concepts. And among all the doubts that have arisen, the main one is about data "duplication".

For example, a situation that I'm creating for learning, as it reflects a scenario from the company where I work.

I a DW concept with 3 stages: raw (raw data), preparation (processed data, with some enrichment, code replacement for code description, formats, etc.) and production (contains fact and dimension tables, which will serve as data sources for PowerBi dashboards).

The doubt is about these 3 stages and how data is duplicated as it passes through them. And given my lack of knowledge, it seems like a serious waste (or at least misuse) of space. Since I have the raw data in the raw layer, which is consolidated, enriched, converted into some formats, but is basically the same thing, and the biggest difference is in the production layer, where I have the cross-referenced data, fact and dimension tables.

It gives the impression that the preparation layer is transitory, therefore disposable, does that make sense?


r/datawarehouse • • Mar 10 '25

Uc berkeley student conducting Fabric research

4 Upvotes

Hey everyone! UC Berkeley student here studying cog sci. I'm conducting user research on Microsoft Fabric for my Data Science class and looking to connect with people who have experience using it professionally or personally.

Please pm if u have!!!


r/datawarehouse • • Aug 09 '23

Ultimate dbt Cheat Sheet

Thumbnail
datacoves.com
4 Upvotes

r/datawarehouse • • Aug 01 '23

Data Warehouse Career switch

4 Upvotes

I am currently a senior .NET developer with 30+ years experience. My entire career I have worked with databases and am researching making the career switch to Data Warehousing. I have built my current companies small data warehouse from the ground up using SQL server and C#. I am proficient in SQL server but will be taking the Maven Analytics SQL courses. I have ordered the The Data Warehouse Toolkit by Kimball and will be reading that. I have experience in Unix but my experience is dated. I'm trying to get any weaknesses I have shored up before looking for a new position. A month ago I didn't know what ETL was and now know that I am doing ETL and ELT in the current data warehouse but industry acronyms and buzzwords are definitely a weakness. I feel my SQL skills are fine and am confident I can learn anything I need to make the switch. That being said I don't know where to learn what I should know. I've seen Linux, Python, Snowflake, etc. I know IBM has a data warehousing certificate on Coursera and Coursera has their own beginner level data warehousing course. I need to learn what I don't know and any suggestions on where to start learning it would be great.


r/datawarehouse • • Jul 10 '23

What is a Data Warehouse and why is it Important?

Thumbnail
dasca.org
4 Upvotes

r/datawarehouse • • May 08 '23

Snowflake Certifications—Which One is Best to Pursue in 2023?

Thumbnail
chaosgenius.io
4 Upvotes

r/datawarehouse • • Mar 17 '23

Data Warehouse Architecture and Design: A Reflective Guide

Thumbnail
dasca.org
4 Upvotes

r/datawarehouse • • Feb 20 '23

Starter

4 Upvotes

Can someone give me some advice for starting with modeling data warehouse. Books, course, tutorials??