Which tool is best suited for data transformation in Fabric when dealing with large-scale data that will continue to grow?

• A. Dataflows (Gen2)
• B. Pipelines
• C. SQL-based ETL scripts
• D. Notebooks

Answer: (D)

When data transformation needs to handle growing, evolving volumes and complex logic, notebooks offer the most flexible and scalable approach in Fabric. A notebook lets you encode transformation logic in code (Python, SQL, Scala, etc.), empowering you to implement custom, iterative transformations that adapt quickly as data grows or schemas change. It runs on Fabric’s distributed compute, so you can process large datasets in parallel, use Spark-style workloads, and scale resources up or down as needed. This makes notebooks well suited for ongoing growth, advanced transformations, feature engineering, or integrating data science steps into the ETL flow.

Dataflows Gen2 are excellent for repeatable, low-code ETL and rapid data prep, but they can be less flexible for highly custom or evolving transformation logic. Pipelines focus on orchestration and scheduling of tasks rather than the transformation itself. SQL-based ETL scripts are powerful for SQL-centric tasks but may struggle with complex procedural logic or dynamic changes. Notebooks, by contrast, provide the code-driven flexibility you need to adapt transformations as data continues to grow.