Etl on Hi, I'm Muhammad Amal

Etl on Hi, I'm Muhammad Amal https://muhammadamal.my.id/tags/etl/ Recent content in Etl on Hi, I'm Muhammad Amal Hugo en-us Fri, 08 Aug 2025 09:00:00 +0700 Orchestrating Complex Enterprise Data Syncs with n8n https://muhammadamal.my.id/blog/orchestrating-complex-enterprise-data-syncs-with-n8n/ Fri, 08 Aug 2025 09:00:00 +0700 https://muhammadamal.my.id/blog/orchestrating-complex-enterprise-data-syncs-with-n8n/ How to build n8n data syncs that survive production, with idempotency keys, watermark-based pulls, fan-out fan-in, and a real dead letter queue. October Retro, ETL Pragmatism https://muhammadamal.my.id/blog/october-retro-etl/ Mon, 31 Oct 2022 09:00:00 +0700 https://muhammadamal.my.id/blog/october-retro-etl/ October retro: lightweight ETL shipped. What worked, what bounced. Monitoring ETL Pipelines, Lag, Errors, Throughput https://muhammadamal.my.id/blog/etl-monitoring-lag-errors/ Fri, 28 Oct 2022 09:00:00 +0700 https://muhammadamal.my.id/blog/etl-monitoring-lag-errors/ ETL monitoring: lag, row counts, errors, throughput. Metrics + dashboards + alerts. Backfilling Historical Data Safely https://muhammadamal.my.id/blog/etl-backfill-historical/ Wed, 26 Oct 2022 09:00:00 +0700 https://muhammadamal.my.id/blog/etl-backfill-historical/ Backfill safely: chunked, rate-limited, parallel, merge with live data. Postgres Bulk Loading, COPY and Prepared Statements https://muhammadamal.my.id/blog/postgres-bulk-loading-copy/ Mon, 24 Oct 2022 09:00:00 +0700 https://muhammadamal.my.id/blog/postgres-bulk-loading-copy/ Postgres bulk loading: COPY beats INSERT 100x. UNLOGGED, index management, fast ETL writes. Kafka as a Sync Backbone https://muhammadamal.my.id/blog/kafka-sync-backbone/ Wed, 19 Oct 2022 09:00:00 +0700 https://muhammadamal.my.id/blog/kafka-sync-backbone/ Kafka 3.2 as sync backbone: topic-per-table, consumers, when it pays off vs simpler tools. Schema Drift, Handling Source Changes Without Breaking Pipelines https://muhammadamal.my.id/blog/etl-schema-drift/ Mon, 17 Oct 2022 09:00:00 +0700 https://muhammadamal.my.id/blog/etl-schema-drift/ ETL schema drift: detect, additive vs breaking, flexible patterns that don’t lose data. Idempotent Pipelines and Watermarks https://muhammadamal.my.id/blog/etl-idempotent-watermarks/ Fri, 14 Oct 2022 09:00:00 +0700 https://muhammadamal.my.id/blog/etl-idempotent-watermarks/ Idempotent ETL: watermarks, upsert dedup, exactly-once, double-write failure modes. Building an ETL Pipeline in Go https://muhammadamal.my.id/blog/go-etl-pipeline/ Wed, 12 Oct 2022 09:00:00 +0700 https://muhammadamal.my.id/blog/go-etl-pipeline/ Go ETL: pgx, COPY bulk, concurrent ETL stages. When Go beats Python. Building an ETL Pipeline in Python https://muhammadamal.my.id/blog/python-etl-pipeline/ Mon, 10 Oct 2022 09:00:00 +0700 https://muhammadamal.my.id/blog/python-etl-pipeline/ Python ETL: SQLAlchemy, Polars, BigQuery client. Incremental sync, errors, monitoring. Postgres Logical Replication for CDC https://muhammadamal.my.id/blog/postgres-logical-replication-cdc/ Fri, 07 Oct 2022 09:00:00 +0700 https://muhammadamal.my.id/blog/postgres-logical-replication-cdc/ Postgres logical replication for CDC: publications, slots, wal2json / pgoutput consumption. CDC vs Polling, Choosing Your Sync Strategy https://muhammadamal.my.id/blog/cdc-vs-polling/ Wed, 05 Oct 2022 09:00:00 +0700 https://muhammadamal.my.id/blog/cdc-vs-polling/ CDC vs polling for data sync. When each wins, ops trade-offs, hybrid approach. Why Lightweight ETL Beats Big Data Tools for Most Teams https://muhammadamal.my.id/blog/lightweight-etl-vs-big-data/ Mon, 03 Oct 2022 09:00:00 +0700 https://muhammadamal.my.id/blog/lightweight-etl-vs-big-data/ Lightweight ETL: Python/Go beats Spark/Airflow for most teams. When ‘big data’ is overkill.