3rd Floor, Ami-Hall, 1-1-3 Shibuya,
Shibuya-ku, Tokyo, 150-0002,
Japan
This project explores how a Medallion architecture can be designed using Stack Overflow survey data on Azure Databricks.
The implementation focuses on separating responsibilities between Python-based processing and SQL-based transformations.
Building on this architecture, developer trends from Stack Overflow survey data are analyzed to extract actionable insights.
Within the filtered dataset (countries with ≥5,000 respondents, excluding under 18 and unknown age groups), management roles show the highest average compensation overall.
Among technical roles, Data/AI and Infrastructure rank among the higher-paying tracks in aggregate.
At the industry level, Financial Services, Healthcare, and Media & Advertising tend to exhibit higher compensation levels compared to other sectors.
However, year-by-year analysis indicates that these rankings are not fully stable over time.
Responsibilities are intentionally separated between Python and SQL to improve maintainability and clarity in the Medallion architecture.
Python handles flexible preprocessing, while SQL focuses on stable aggregation logic close to the data layer.
