Medallion Architecture in Practice: Six Layers, One Source of Truth - Incentius Blog Home - Incentius - Enable Digital Transformation [![Logo](/img_astro/incentius-logo.svg)](/) Solutions About [Case Studies](/portfolio/) [Blogs](/blog-posts/) [Career](http://aurochs.freshteam.com/jobs) Open main menu ![Menu icon](/img_astro/menu1.png) [Get In Touch](/contact/) Solutions [![](/img_astro/generative-black.svg) ![](/img_astro/brain-green.svg) Voice AI](/voice-ai/) [![](/img_astro/product-development-black.svg) ![](/img_astro/product-development.svg) Product Development](/product-engineering/) [![](/img_astro/generative-black.svg) ![](/img_astro/brain-green.svg) Generative AI](/generative-ai/) [![](/img_astro/data-black.svg) ![](/img_astro/database-green.svg) Data Engineering](/data-engineering/) [![](/img_astro/mvp-black.svg) ![](/img_astro/rocket-green.svg) Data & Analytics](/data-analytics/) [![](/img_astro/cloud-black.svg) ![](/img_astro/computing-green.svg) DevOps & CloudOps](/devops-and-cloudops/) About Us [![](/img_astro/product-black.svg) ![](/img_astro/product-green.svg) About Us](/about-us/) [![](/img_astro/team_black.svg) ![](/img_astro/team_green.svg) Team](/team/) Solutions ![](/img_astro/dropdown-arr.svg) [ ![](/img_astro/generative-black.svg) Voice AI ](/voice-ai/)[ ![](/img_astro/product-development-black.svg) Product Development ](/product-engineering/)[ ![](/img_astro/generative-black.svg) Generative AI ](/generative-ai/)[ ![](/img_astro/data-black.svg) Data Engineering ](/data-engineering/)[ ![](/img_astro/mvp-black.svg) Data & Analytics ](/data-analytics/)[ ![](/img_astro/cloud-black.svg) DevOps & CloudOps ](/devops-and-cloudops/) [Blogs](/blog-posts/) About ![](/img_astro/dropdown-arr.svg) [ ![](/img_astro/product-black.svg) About Us ](/about-us/)[ ![](/img_astro/team_black.svg) Team ](/team/) [Case Studies](/portfolio/) [Career](http://aurochs.freshteam.com/jobs) [Get In Touch](/contact/) Medallion Architecture in Practice: Six Layers, One Source of Truth Dilkesh Tilokchandani 1. Jun 04, 2026 2. 4 min read ![](/_astro/image_uuuDNhO.width-1000_uPMRi.webp) Two hundred source definitions. Forty-seven job pipelines. Five hundred and sixteen transformation models. And not a single bespoke DAG written for any of them. That's the headline result of a data platform we built to ingest, transform, and distribute healthcare and commercial data at scale - all driven by YAML configuration rather than hand-written pipeline code. Here's the architecture behind it, and what we're proud of delivering. ## **The Core Idea: Configuration Over Code** The guiding principle behind this platform is simple: adding a new data source should mean writing a YAML file, not a new DAG. Instead of writing logic per source, we built general-purpose DAG generators that read metadata and produce the right Airflow behavior dynamically. The YAML files are the source of truth. This single decision shaped everything else about the platform. ## **Architecture Overview** The platform has two connected repositories working in tandem: an orchestration layer built on Airflow, and a transformation layer built on dbt. Together they form a clean end-to-end pipeline - from raw vendor files to business-ready outputs distributed to downstream consumers. ### **The Ingestion Layer** For each data source, a YAML file defines everything the system needs to know: where the file comes from (SFTP, S3, Salesforce SOQL, or an external API), the filename pattern, load type, delimiter and encoding, archive behavior, and column-level data quality expectations. A DAG generator reads these files at runtime and produces one Airflow DAG per source. Each DAG follows a consistent, reliable sequence: 1. Fetch files from the source system 2. Land them in an S3 input bucket 3. Log ingestion status to a Postgres/RDS metadata store 4. Load into Snowflake raw tables using COPY INTO 5. Run dbt tests on raw data 6. Build base models 7. Run dbt tests again 8. Archive processed files Sensitive datasets follow a dedicated PII-specific path that routes them into an isolated Snowflake schema with separate bucket and stage configurations - keeping compliance requirements cleanly separated from the main pipeline with no extra work from the engineers adding new sources. ### **The Medallion Architecture: Six Layers of Trust** The heart of the transformation layer is a medallion-style architecture implemented in dbt. Rather than consolidating everything into a single schema and hoping for the best, we designed six distinct layers - each with a clear purpose, increasing levels of refinement, and explicit contracts between them. The idea is straightforward: an engineer can look at any model path and instantly know how mature and trustworthy the data is. A raw\_ prefix signals unprocessed ingestion straight from source. A reporting\_ prefix signals a polished, validated, business-approved output. The layers in between are the journey from one to the other. **RAW** is the landing zone. Data arrives here exactly as it came from the source - no transformations, no business logic. Every source table is preserved in its original form, giving a full audit trail and the ability to reprocess from scratch if needed. **BASE** is where raw data gets its first treatment. Light cleaning happens here: standardising column names, enforcing data types, trimming whitespace, handling nulls, and normalising encodings. BASE models are thin but consistent - they create a reliable foundation for everything built on top. **CORE** is where the real transformation happens. This layer joins tables, deduplicates records, applies business rules, and produces the canonical representation of each business entity. If BASE is about technical correctness, CORE is about business correctness. A model like core\_patients or core\_prescriptions reflects what those entities actually mean to the organisation - not just what arrived in a file. CORE is the truth layer the entire downstream stack is built on. **ANALYTICS** builds on CORE to produce aggregated, metric-level models - pre-computed measures, period-over-period comparisons, cohort rollups. By materialising these in the warehouse rather than in BI tools, every consumer querying the data sees consistent numbers. **REPORTING** packages analytics outputs into consumption-ready datasets shaped to the exact needs of specific consumers - dashboards, stakeholder reports, or downstream applications. These models are denormalised and labelled with business-friendly names, requiring minimal transformation by the end consumer. **FORECASTING** is the specialised layer for predictive and modelling workflows. It draws from CORE and ANALYTICS to feed statistical models, demand forecasts, and scenario analyses - keeping predictive logic cleanly separated from the operational reporting stack. ![](/_astro/image_1.width-1000_PTJq2.webp) ### **Job Orchestration** Job orchestration reads job YAML files to create sequential Airflow DAGs. A job can trigger source DAGs, trigger other jobs, run dbt commands, wait on external tasks, capture dbt artifacts, and send status or data quality emails. This composability is what makes the whole system feel like one cohesive platform rather than a collection of independent pipelines. ### **Outbound Publishing** Data doesn't just flow in — it flows out. A feeds framework reads feed YAML files, executes Snowflake queries, generates output files, and uploads them to S3 or pushes directly back to Salesforce. A separate file transfer manager handles standalone SFTP-to-S3 delivery for vendor partners. ## **What We Achieved** **Scale without complexity.** The platform handles over 200 source YAML definitions, 47 job definitions, 15 feed definitions, and around 516 SQL models - spanning commercial data domains including Veeva, Prognos, Claritas, SFMC, MMIT, Komodo, Cardinal 3PL, and others. Adding a new vendor typically involves writing a source YAML and a handful of dbt models. No new DAG code, no new operators. **Operational reliability built in.** Every source pipeline - without exception - gets ingestion logging, file archiving, DQ test gates, and dbt artifact capture. These aren't features someone has to remember to add, they're part of the framework that every YAML-defined source inherits automatically. **Compliance handled gracefully.** PII data flows through a fully isolated path with separate infrastructure, enforced by the framework - not by individual engineers remembering to do the right thing. **A trustworthy data contract across six layers.** The medallion architecture gives every downstream consumer a clear signal about what they're working with. RAW preserves the original record. BASE standardises it. CORE establishes business truth. ANALYTICS aggregates it consistently. REPORTING delivers it consumption-ready. FORECASTING powers predictive work, cleanly separated from operational flows. Each layer earns its name. **Real-world edge cases solved.** ZIP file ingestion, incremental file detection, encoding variations, outbound Salesforce syncs, custom QC reporting - these aren't afterthoughts. They're built into the framework as first-class capabilities, accumulated through the real experience of working with dozens of external vendors. ## **The Bigger Picture** The metadata-driven approach isn't new - it's a well-established pattern in data engineering. What made it work here was the discipline to commit to it consistently: across ingestion, transformation, orchestration, and outbound publishing, rather than applying it selectively and falling back to bespoke code when things got complicated. The result is a platform that scales with the business. New data sources, new business domains, new downstream consumers - all handled through configuration. The engineering effort goes into improving the framework, not repeating the same work for the hundredth source. For teams considering a similar path, our advice: invest in the YAML schema design before writing a single line of generator code. The shape of your configuration files determines everything downstream, and it's far easier to get that right at the start than to migrate 200 source definitions later. _Want to learn more about how we approach data platform engineering? Reach out to our team - we'd love to hear how you're tackling similar challenges._ About Author ![](/_astro/dilkesh.width-300_xNiBl.webp) Dilkesh Tilokchandani ![](/img_astro/thumb.png) ### See What Our Clients Say ![](/img_astro/left-chevron_1.svg) ![](/img_astro/left-chevron_2.svg) ![](/img_astro/mind-gap-crop.svg) Mindgap Incentius has been a fantastic partner for us. Their strong expertise in technology helped deliver some complex solutions for our customers within challenging timelines. Specific call out to Sujeet and his team who developed custom sales analytics dashboards in SFDC for a SoCal based healthcare diagnostics client of ours. Their professionalism, expertise, and flexibility to adjust to client needs were greatly appreciated. MindGap is excited to continue to work with Incentius and add value to our customers. Samik Banerjee Founder & CEO ![](/img_astro/trusted-by-businesses/worldatwork_coloured.webp) World at Work Having worked so closely for half a year on our website project, I wanted to thank Incentius for all your fantastic work and efforts that helped us deliver a truly valuable experience to our WorldatWork members. I am in awe of the skills, passion, patience, and above all, the ownership that you brought to this project every day! I do not say this lightly, but we would not have been able to deliver a flawless product, but for you. I am sure you'll help many organizations and projects as your skills and professionalism are truly amazing. Shantanu Bayaskar Senior Project Manager ![](/img_astro/trusted-by-businesses/gogla_coloured.webp) Gogla It was a pleasure working with Incentius to build a data collection platform for the off-grid solar sector in India. It is rare to find a team with a combination of good understanding of business as well as great technological know-how. Incentius team has this perfect combination, especially their technical expertise is much appreciated. We had a fantastic time working with their expert team, especially with Amit. Viraj gada Gogla ![](/img_astro/trusted-by-businesses/humblex_coloured.webp) Humblx Choosing Incentius to work with is one of the decisions we are extremely happy with. It's been a pleasure working with their team. They have been tremendously helpful and efficient through the intense development cycle that we went through recently. The team at Incentius is truly agile and open to a discussion in regards to making tweaks and adding features that may add value to the overall solution. We found them willing to go the extra mile for us and it felt like working with someone who rooted for us to win. Samir Dayal Singh CEO Humblx ![](/img_astro/mind-gap-crop.svg) Transportation & Logistics Consulting Organization Incentius is very flexible and accommodating to our specific needs as an organization. In a world where approaches and strategies are constantly changing, it is invaluable to have an outsourcer who is able to adjust quickly to shifts in the business environment. Transportation & Logistics Consulting Organization Consultant ![]() Mudraksh & McShaw Incentius was instrumental in bringing the visualization aspect into our investment and trading business. They helped us organize our trading algorithms processing framework, review our backtests and analyze results in an efficient, visual manner. Priyank Dutt Dwivedi Mudraksh & McShaw Advisory ![](/img_astro/mind-gap-crop.svg) Leading Healthcare Consulting Organization The Incentius resource was highly motivated and developed a complex forecasting model with minimal supervision. He was thorough with quality checks and kept on top of multiple changes. Leading Healthcare Consulting Organization Sr. Principal ![](/img_astro/mind-gap-crop.svg) US Fortune 100 Telecommunications Company The Incentius resource was highly motivated and developed a complex forecasting model with minimal supervision. He was thorough with quality checks and kept on top of multiple changes. Incentive Compensation Sr. Director Most Read [ ![](/_astro/ChatGPT_Image_Apr_2_2025_12_06_35_PM.width-1000_ZUChn3.webp) Optimizing Large-Scale Data Exports: FastAPI, GraphQL, and AWS S3 Struggling with slow, inefficient file downloads in your application? Generating and handling large file downloads can be a nightmare—UI freezing, server overload, and endless waiting. ![](/_astro/Vinay_Chaudhari.width-300_1QPJnB.webp) Vinay Chaudhari 1. Apr 08, 2025 2. 4 min read ](/blog-posts/optimizing-large-scale-data-exports-fastapi-graphql-and-aws-s3/)[ ![](/_astro/Screenshot_2025-04-02_at_12.09.33_PM.width-1000_ZyU6y1.webp) How to Build a Web Search Agent with Phidata: A Step-by-Step Guide Phidata is an **open-source framework** for building AI agents with capabilities like memory, tool use, and reasoning​ It’s model-agnostic and supports various large language models (LLMs), allowing developers to turn any LLM (like OpenAI GPT or local models) into a functional **search agent**. ![](/_astro/Kartik_Puri.width-300_1YNCyp.webp) Kartik Puri 1. Apr 02, 2025 2. 4 min read ](/blog-posts/how-to-build-a-web-search-agent-with-phidata-a-step-by-step-guide/)[ ![](/_astro/e20f5335-cbf6-414c-a096-7d2849109cff.width-1000_15DfBS.webp) Migrating to Soketi: Improving Our Real- Time Communication Architecture Our journey with real-time data communication has evolved significantly. Initially using Flask-SocketIO with Vue, we faced several challenges that led us to seek a more robust solution. ![](/_astro/pratik.width-300_Z2vB9eu.webp) Pratik Patel 1. Mar 17, 2025 2. 4 min read ](/blog-posts/migrating-to-soketi-improving-our-real-time-communication-architecture/)[ ![](/_astro/startup-employee-looking-business-charts-usin.width-1000_2knrAp.webp) Understanding AI Agents: A Comprehensive Guide to AI Agentic Workflows In the rapidly evolving landscape of artificial intelligence, AI Agents have emerged as a transformative technology that represents far more than just industry jargon. ![](/_astro/chetan_piucv.width-300_ZK3Vor.webp) Chetan Patel 1. Feb 28, 2025 2. 4 min read ](/blog-posts/understanding-ai-agents-a-comprehensive-guide-to-ai-agentic-workflows/) ## We use cookies We use cookies to improve site. Some cookies are necessary for our website and services to function properly. Other cookies are optional and help personalize your experience, including advertising and analytics.You can consent to all cookies or decline all optional cookies. Without a selection, our default cookie settings will apply. I agree I decline [![logo](/img_astro/logo.svg)](/) Incentius is a technology and professional services company providing analytics, operations and product engineering services. [![facebook](/img_astro/facebook.svg)](https://www.facebook.com/Incentius/)[![twitter](/img_astro/twitter.svg) ](https://x.com/incentius/)[![linkedin](/img_astro/linkedin.svg)](https://www.linkedin.com/company/incentius/posts/?feedView=all) ![location](/img_astro/pin.svg) Office No.201B, 2nd Floor, Beta-1, Giga Space IT Park, Viman Nagar, Pune - 411014, Maharashtra, India ![email](/img_astro/message-icon.svg) info@incentius.com ![phone](/img_astro/telephone.svg) +1 (302) 415 3208 ![](/img_astro/secure-footer.svg) ![](/img_astro/secure-footer-mobile.svg) * Solutions * [Voice AI](/voice-ai/) * [Generative AI](/generative-ai/) * [Data Engineering](/data-engineering/) * [Product Development](/product-engineering/) * [Data & Analytics](/data-analytics/) * [DevOps & CloudOps](/devops-and-cloudops/) * [UI/UX Design](/ui-ux-design/) * Recent Blog * [Medallion Architecture in Practice: Six Layers, One Source of Truth](/blog-posts/medallion-architecture-in-practice-six-layers-one-source-of-truth/) * ["Document later" is how teams say "document never"](/blog-posts/document-later-is-how-teams-say-document-never/) * [Your Business & AI - What It Actually Means for You](/blog-posts/your-business-ai-what-it-actually-means-for-you/) * [How Internal Agent-Built Software Reduced Our SaaS Dependency](/blog-posts/how-internal-agent-built-software-reduced-our-saas-dependency/) * [MarkItDown: A Powerful Utility for File Conversion to Markdown](/blog-posts/markitdown-a-powerful-utility-for-file-conversion-to-markdown/) * [Python 3.14: A Game-Changing Release for Modern Development](/blog-posts/python-314-a-game-changing-release-for-modern-development/) * [Building a RESTful Blog API with Flask and Swagger: A Complete Guide to Streamlined API Development](/blog-posts/building-a-restful-blog-api-with-flask-and-swagger-a-complete-guide-to-streamlined-api-development/) * [Tailwind CSS 4.0 vs 3.x: What Changed and How We Implemented It](/blog-posts/tailwind-css-40-vs-3x-what-changed-and-how-we-implemented-it/) * [The Future of AI: What to Expect in the Next 5 Years](/blog-posts/the-future-of-ai-what-to-expect-in-the-next-5-years/) * Menu * [About](/about-us/) * [Blog](/blog-posts/) * [Career](http://aurochs.freshteam.com/jobs) * [Case Studies](/portfolio/) * [For Startups](/startups/) * [Let's Talk](/contact/) © Copyright 2025 Incentius [Request Site](/) [Terms and Conditions](/) [Privacy Policy](/privacy-policy/)