byte_array's profile picture. Founder @Onehousehq, Creator of @apachehudi, Built the World's first #DataLakehouse, Distributed/Data Systems, Linkedin, Uber, Confluent alum. (views are mine)

Vinoth Chandar

@byte_array

Founder @Onehousehq, Creator of @apachehudi, Built the World's first #DataLakehouse, Distributed/Data Systems, Linkedin, Uber, Confluent alum. (views are mine)

Fijado

🔥 Meet Quanton — the new query execution engine from Onehouse. 👍 Same Spark & SQL. 📉 At least half the cost. 📈 1.6x-3.6x better ETL price-performance 📊 2.2x-6.5x better Ingest price-performance 👉  Read the full blog here: onehouse.ai/blog/announcin… ⬇️  Download our free…


💰🔥 Spark’s default autoscaler = higher latency scaling up, wasted $$ scaling down. Why? It’s based on task backlog, not actual resource usage. Costly flaw. Result: 🐢 Slow scale-ups (e.g. too few Kafka partitions during spikes) 🐢 Slow scale-downs (e.g. many tiny tasks →…

byte_array's tweet image. 💰🔥 Spark’s default autoscaler = higher latency scaling up, wasted $$ scaling down.

Why? It’s based on task backlog, not actual resource usage. Costly flaw.

Result:
🐢 Slow scale-ups (e.g. too few Kafka partitions during spikes)
🐢 Slow scale-downs (e.g. many tiny tasks →…

📊 If you’re using Apache Spark on EMR (or anywhere really), you need better visibility into where your compute spend is going. At Onehouse, we kept seeing the same pattern across Hudi and non-Hudi users alike: 👉 Jobs were under-optimized 👉 Executors were sitting idle 👉…


🗞️ OLD NEWS: but worth a shout-out. Keys are optional in Hudi ... One of Hudi’s core goals was to remove friction from building data lakes. That’s why — for a long time now — Hudi has quietly supported auto-generation of record keys. No need to think up a key field just to get…

byte_array's tweet image. 🗞️ OLD NEWS: but worth a shout-out. Keys are optional in Hudi ...

One of Hudi’s core goals was to remove friction from building data lakes. That’s why — for a long time now — Hudi has quietly supported auto-generation of record keys. No need to think up a key field just to get…

Loading...

Something went wrong.


Something went wrong.