Real-time analytics databases, #ApacheBeam and HPE Greenlake's new file storage are in my #bigdata weekly udpate


Watching the amazing @rmehmandarov build data pipelines w/ #java #opensource #apachebeam #gcp w/ windowing transformation aggregation #streaming #batch

mgrygles's tweet image. Watching the amazing @rmehmandarov build data pipelines w/ #java #opensource #apachebeam #gcp w/ windowing transformation aggregation #streaming #batch
mgrygles's tweet image. Watching the amazing @rmehmandarov build data pipelines w/ #java #opensource #apachebeam #gcp w/ windowing transformation aggregation #streaming #batch
mgrygles's tweet image. Watching the amazing @rmehmandarov build data pipelines w/ #java #opensource #apachebeam #gcp w/ windowing transformation aggregation #streaming #batch

I'm excited to share that I'll be a mentor at the Beam College Hackathon alongside the amazing people from Google!🤩 Join us May 15–18 for an exciting deep dive into Apache Beam, Google Cloud Dataflow, AI-powered data processing, and more fun stuff! #dataengineering #apachebeam

thecodemancer_'s tweet image. I'm excited to share that I'll be a mentor at the Beam College Hackathon alongside the amazing people from Google!🤩

Join us May 15–18 for an exciting deep dive into Apache Beam, Google Cloud Dataflow, AI-powered data processing, and more fun stuff!

#dataengineering #apachebeam

🎊🥳What an absolute blast! I'm beyond excited and incredibly thankful to have received a Certificate of Appreciation for my time as a mentor at Beam College's Hackathon 2025! 🎉🚀 #dataengineering #apachebeam

thecodemancer_'s tweet image. 🎊🥳What an absolute blast! I'm beyond excited and incredibly thankful to have received a Certificate of Appreciation for my time as a mentor at Beam College's Hackathon 2025! 🎉🚀

#dataengineering #apachebeam

Tried out #ApacheAirflow tonight. Its Python based and smooth enough setup. The UI is nice for making workflows in DAGs. Seems perfect for scheduled/batch jobs ⏰ Going to dive into #ApacheBeam next to try some real-time streaming

gortron's tweet image. Tried out #ApacheAirflow tonight. Its Python based and  smooth enough setup. The UI is nice for making workflows in DAGs. Seems perfect for scheduled/batch jobs ⏰ Going to dive  into #ApacheBeam next to try some real-time streaming

A question that’s been echoing in the minds of many data engineers — “Why don’t we see more adoption of Apache Beam in the industry?” 🤔 ⬇️ Here are a few compelling reasons why one should choose Apache Beam: 🧵[1/2] #dataengineering #ApacheBeam #datascience #MachineLearning

thecodemancer_'s tweet image. A question that’s been echoing in the minds of many data engineers — “Why don’t we see more adoption of Apache Beam in the industry?” 🤔

⬇️ Here are a few compelling reasons why one should choose Apache Beam:

🧵[1/2]

#dataengineering #ApacheBeam #datascience #MachineLearning

Do you know exactly when you should use #ApacheFlink or #ApacheBeam in your solutions?🤔 If you don't, join me at the @AmsterdamJug on November 2nd to find the answers! 🚀 meetup.com/amsterdam-java… #AmsterdamJUG #StreamProcessing #Dataflow #Apache

sorooshkh's tweet image. Do you know exactly when you should use #ApacheFlink or #ApacheBeam in  your solutions?🤔
If you don't, join me at the @AmsterdamJug on November  2nd to find the answers! 🚀 
meetup.com/amsterdam-java…

 #AmsterdamJUG #StreamProcessing #Dataflow #Apache

The Akvelon team is thrilled to collaborate with the @ApacheBeam community and proudly presents ‘Tour of Beam’ - a comprehensive and interactive learning guide that streamlines your #ApacheBeam learning experience. Embark on your tour now at: lnkd.in/eFyjzBJF. Try Tour…


Apache Flink is the best choice for streaming, right? 🚀 But what if I told you there’s a way to make it even more flexible? 👇 🧵[1/X] #DataEngineering #ApacheFlink #ApacheBeam #ETL #DataScience #Tech #Programming

thecodemancer_'s tweet image. Apache Flink is the best choice for streaming, right? 🚀 But what if I told you there’s a way to make it even more flexible?

👇

🧵[1/X]  

#DataEngineering #ApacheFlink #ApacheBeam #ETL #DataScience #Tech #Programming

Apache Hop 2.5.0 is available! 🎯 work on 100 tickets 🎯 #ApacheBeam upgrade to 2.48.0 🎯 new how-to guides 🎯 new Intersystems IRIS database type 🎯 Hop Gui config improvements 🎯 community growth know-bi.be/blog/apache-ho… #apachehop #dataengineering #dataorchestration #knowbi

bartmaer's tweet image. Apache Hop 2.5.0 is available!

🎯 work on 100 tickets
🎯 #ApacheBeam upgrade to 2.48.0
🎯 new how-to guides
🎯 new Intersystems IRIS database type
🎯 Hop Gui config improvements
🎯 community growth

know-bi.be/blog/apache-ho…

#apachehop #dataengineering #dataorchestration
#knowbi

Quiet before the storm – the "backstage view". 5 minutes later the room was packed! 🤩 Just done presenting on data pipelines, #apachebeam, #OSS, #Java & #Cloud @ Devfest Bucharest, on the Cloud stage. 💾 Thanks to the awesome @GDGBucharest team for the organization. 👏

rmehmandarov's tweet image. Quiet before the storm – the "backstage view". 5 minutes later the room was packed! 🤩

Just done presenting on data pipelines, #apachebeam, #OSS, #Java & #Cloud @ Devfest Bucharest, on the Cloud stage. 💾

Thanks to the awesome @GDGBucharest team for the organization. 👏

🚀 We're at #BeamSummit in NYC! 🎤 ML6 Engineer Konstantin just shared how agents can build & run Beam pipelines in his talk: "Superpowering Agents with #ApacheBeam". 📽️ Stay tuned for recordings: hubs.la/Q03wPmr40

ml6team's tweet image. 🚀 We're at #BeamSummit in NYC!
🎤 ML6 Engineer Konstantin just shared how agents can build & run Beam pipelines in his talk: "Superpowering Agents with #ApacheBeam".
📽️ Stay tuned for recordings: hubs.la/Q03wPmr40

“Beam me up, Scotty!” Okay, this is a different Beam—but it’s just as useful. #ApacheBeam is a versatile #opensource framework for designing and executing #data processing pipelines. Now, imagine you’re building an e-commerce app that captures product orders and customer…

redpandadata's tweet image. “Beam me up, Scotty!”

Okay, this is a different Beam—but it’s just as useful. 

#ApacheBeam is a versatile #opensource framework for designing and executing #data processing pipelines. Now, imagine you’re building an e-commerce app that captures product orders and customer…

Want to set up an ETL pipeline using Cloud Dataflow & Apache Beam quickly? Learn how with our step-by-step tutorial! 🛠 Discover installation steps, FAQs, and more! 🔗 Read now: buff.ly/3Q2KUde #GCPDataflow #ApacheBeam #ETLPipeline #CloudCosts

economize's tweet image. Want to set up an ETL pipeline using Cloud Dataflow & Apache Beam quickly? 

Learn how with our step-by-step tutorial!

🛠 Discover installation steps, FAQs, and more!

🔗 Read now: buff.ly/3Q2KUde 

#GCPDataflow #ApacheBeam #ETLPipeline #CloudCosts

Meanwhile, Hadoop, Spark and Flink evolved in parallel around same timeline #apachebeam #flink #spark


Day 9/180 Wrapping up Apache Beam 🚀 ✔️ Applied all core + advanced streaming concepts ✔️ Built an end-to-end use case to reinforce learning Next up: new tools, new challenges 🔥 #180DaysOfDE #DataEngineering #ApacheBeam


🚀 We're at #BeamSummit in NYC! 🎤 ML6 Engineer Konstantin just shared how agents can build & run Beam pipelines in his talk: "Superpowering Agents with #ApacheBeam". 📽️ Stay tuned for recordings: hubs.la/Q03wPmr40

ml6team's tweet image. 🚀 We're at #BeamSummit in NYC!
🎤 ML6 Engineer Konstantin just shared how agents can build & run Beam pipelines in his talk: "Superpowering Agents with #ApacheBeam".
📽️ Stay tuned for recordings: hubs.la/Q03wPmr40

The #GoogleCloud #Dataflow now supports right fitting — this is a new feature that leverages #ApacheBeam resource hints to customize resource usage within a data pipeline.


🎊🥳What an absolute blast! I'm beyond excited and incredibly thankful to have received a Certificate of Appreciation for my time as a mentor at Beam College's Hackathon 2025! 🎉🚀 #dataengineering #apachebeam

thecodemancer_'s tweet image. 🎊🥳What an absolute blast! I'm beyond excited and incredibly thankful to have received a Certificate of Appreciation for my time as a mentor at Beam College's Hackathon 2025! 🎉🚀

#dataengineering #apachebeam

🌟Congratulations to the top 5 participants on our Online Training Sessions leaderboard for their outstanding engagement, and to the Beam College 2025 Hackathon winners for their innovative solutions using #ApacheBeam! Special shoutout to: 🥇 PCollectors 🥈 Warrior 🥉 Anomaflow

beam_college's tweet image. 🌟Congratulations to the top 5 participants on our Online Training Sessions leaderboard for their outstanding engagement, and to the Beam College 2025 Hackathon winners for their innovative solutions using #ApacheBeam!
Special shoutout to:
🥇 PCollectors
🥈 Warrior
🥉 Anomaflow

📢💻 The #BeamCollege Hackathon kicks off TODAY! Now’s your chance to build something amazing using #ApacheBeam and showcase the skills you have learned these past couple of days! Don't miss your shot to win up to $1,500 USD and participate now ⚙️ 👉 beamcollege.dev/hackathon/

beam_college's tweet image. 📢💻 The #BeamCollege Hackathon kicks off TODAY!

Now’s your chance to build something amazing using #ApacheBeam and showcase the skills you have learned these past couple of days! 

Don't miss your shot to win up to $1,500 USD and participate now ⚙️

👉 beamcollege.dev/hackathon/

I'm excited to share that I'll be a mentor at the Beam College Hackathon alongside the amazing people from Google!🤩 Join us May 15–18 for an exciting deep dive into Apache Beam, Google Cloud Dataflow, AI-powered data processing, and more fun stuff! #dataengineering #apachebeam

thecodemancer_'s tweet image. I'm excited to share that I'll be a mentor at the Beam College Hackathon alongside the amazing people from Google!🤩

Join us May 15–18 for an exciting deep dive into Apache Beam, Google Cloud Dataflow, AI-powered data processing, and more fun stuff!

#dataengineering #apachebeam

🚨Beam College 2025 starts in 2 days! Join us online from May 15th to 18th for a free, hands-on event to explore or advance your skills with #ApacheBeam🐝🎓 Get ready for 2 days of training sessions ➕ 2 more days of a dynamic hackathon. 🔗 beamcollege.dev

beam_college's tweet image. 🚨Beam College 2025 starts in 2 days! 

Join us online from May 15th to 18th for a free, hands-on event to explore or advance your skills with #ApacheBeam🐝🎓 

Get ready for 2 days of training sessions ➕ 2 more days of a dynamic hackathon.

🔗 beamcollege.dev

Rakesh Kumar will be a speaker at #BeamCollege 2025! 🌟 🚀 Delve into the diverse functionalities provided by stateful processing and understand how to leverage #ApacheBeam's capabilities effectively in data engineering projects. 👉beamcollege.dev

beam_college's tweet image. Rakesh Kumar will be a speaker at #BeamCollege 2025! 🌟

🚀 Delve into the diverse functionalities provided by stateful processing and understand how to leverage #ApacheBeam's capabilities effectively in data engineering projects.

👉beamcollege.dev

Master new #ApacheBeam features at #BeamCollege 2025🐝 💫 1️⃣ Learn how you can create low-code and no-code Beam YAML jobs with Ryan Madden. 2️⃣ Explore a new way to authorize and run Beam pipelines via the YAML SDK with Svetak Sundhar. Register now: beamcollege.dev

beam_college's tweet image. Master new #ApacheBeam features at #BeamCollege 2025🐝 💫

1️⃣ Learn how you can create low-code and no-code Beam YAML jobs with Ryan Madden. 
2️⃣ Explore a new way to authorize and run Beam pipelines via the YAML SDK with Svetak Sundhar. 

Register now: beamcollege.dev

🟠 Build a GenAI app 🟡 Create a pipeline without coding 🟠 Build a RAG pipeline 🟡 Easily read and write to Apache Iceberg To learn more about the Beam College, visit beamcollege.dev #dataengineering #apachebeam 🧵[4/x]


are then made available on-demand for future reference. The 2025 edition of Beam College also includes a hackathon where participants build data processing solutions and compete for prizes. In this edition you will: #dataengineering #apachebeam 🧵[3/x]


Beam College is an online, free educational program that provides hands-on training to solve data processing use cases using Apache Beam®. The program consists of live online training sessions which ... #dataengineering #apachebeam 🧵[2/x]


🎉 I'm excited to share that I'll be a mentor at the Beam College Hackathon! What is Beam College? #dataengineering #apachebeam 🧵[1/x]

thecodemancer_'s tweet image. 🎉 I'm excited to share that I'll be a mentor at the Beam College Hackathon!

What is Beam College?

#dataengineering #apachebeam

🧵[1/x]

💥 Be part of #BeamCollege 2025! Join Claude van der Merwe as he introduces vector-based semantic search and Retrieval Augmented Generation (#RAG), demonstrating how to build scalable pipelines for using #ApacheBeam. 💻Save your spot now at: beamcollege.dev

beam_college's tweet image. 💥 Be part of #BeamCollege 2025! 

Join Claude van der Merwe as he introduces vector-based semantic search and Retrieval Augmented Generation (#RAG), demonstrating how to build scalable pipelines for using #ApacheBeam. 

💻Save your spot now at: beamcollege.dev

🛠️Israel Herraiz will demonstrates how to perform streaming inference with #ApacheBeam and GoogleAI Studio’s #Gemini model, focusing on a geography-based example to get country capitals. Catch him at #BeamCollege 2025 and add his session to your calendar: beamcollege.dev/sessions/2025/

beam_college's tweet image. 🛠️Israel Herraiz will demonstrates how to perform streaming inference with #ApacheBeam and GoogleAI Studio’s #Gemini model, focusing on a geography-based example to get country capitals.

Catch him at #BeamCollege 2025 and add his session to your calendar: beamcollege.dev/sessions/2025/

Watching the amazing @rmehmandarov build data pipelines w/ #java #opensource #apachebeam #gcp w/ windowing transformation aggregation #streaming #batch

mgrygles's tweet image. Watching the amazing @rmehmandarov build data pipelines w/ #java #opensource #apachebeam #gcp w/ windowing transformation aggregation #streaming #batch
mgrygles's tweet image. Watching the amazing @rmehmandarov build data pipelines w/ #java #opensource #apachebeam #gcp w/ windowing transformation aggregation #streaming #batch
mgrygles's tweet image. Watching the amazing @rmehmandarov build data pipelines w/ #java #opensource #apachebeam #gcp w/ windowing transformation aggregation #streaming #batch

Do you know exactly when you should use #ApacheFlink or #ApacheBeam in your solutions?🤔 If you don't, join me at the @AmsterdamJug on November 2nd to find the answers! 🚀 meetup.com/amsterdam-java… #AmsterdamJUG #StreamProcessing #Dataflow #Apache

sorooshkh's tweet image. Do you know exactly when you should use #ApacheFlink or #ApacheBeam in  your solutions?🤔
If you don't, join me at the @AmsterdamJug on November  2nd to find the answers! 🚀 
meetup.com/amsterdam-java…

 #AmsterdamJUG #StreamProcessing #Dataflow #Apache

Deploy #apachebeam job on #google #dataflow? Use custom container SDK image and scale up to workers in NO time #dataengineering #googlecloud #dataflow #apache #softwareengineering

sounish1's tweet image. Deploy #apachebeam job on #google #dataflow?
Use custom container SDK image and scale up to workers in NO time

#dataengineering #googlecloud #dataflow #apache #softwareengineering

I'm excited to share that I'll be a mentor at the Beam College Hackathon alongside the amazing people from Google!🤩 Join us May 15–18 for an exciting deep dive into Apache Beam, Google Cloud Dataflow, AI-powered data processing, and more fun stuff! #dataengineering #apachebeam

thecodemancer_'s tweet image. I'm excited to share that I'll be a mentor at the Beam College Hackathon alongside the amazing people from Google!🤩

Join us May 15–18 for an exciting deep dive into Apache Beam, Google Cloud Dataflow, AI-powered data processing, and more fun stuff!

#dataengineering #apachebeam

Tried out #ApacheAirflow tonight. Its Python based and smooth enough setup. The UI is nice for making workflows in DAGs. Seems perfect for scheduled/batch jobs ⏰ Going to dive into #ApacheBeam next to try some real-time streaming

gortron's tweet image. Tried out #ApacheAirflow tonight. Its Python based and  smooth enough setup. The UI is nice for making workflows in DAGs. Seems perfect for scheduled/batch jobs ⏰ Going to dive  into #ApacheBeam next to try some real-time streaming

A question that’s been echoing in the minds of many data engineers — “Why don’t we see more adoption of Apache Beam in the industry?” 🤔 ⬇️ Here are a few compelling reasons why one should choose Apache Beam: 🧵[1/2] #dataengineering #ApacheBeam #datascience #MachineLearning

thecodemancer_'s tweet image. A question that’s been echoing in the minds of many data engineers — “Why don’t we see more adoption of Apache Beam in the industry?” 🤔

⬇️ Here are a few compelling reasons why one should choose Apache Beam:

🧵[1/2]

#dataengineering #ApacheBeam #datascience #MachineLearning

🎊🥳What an absolute blast! I'm beyond excited and incredibly thankful to have received a Certificate of Appreciation for my time as a mentor at Beam College's Hackathon 2025! 🎉🚀 #dataengineering #apachebeam

thecodemancer_'s tweet image. 🎊🥳What an absolute blast! I'm beyond excited and incredibly thankful to have received a Certificate of Appreciation for my time as a mentor at Beam College's Hackathon 2025! 🎉🚀

#dataengineering #apachebeam

Apache Beam lets you write one pipeline & execute it on multiple engines (Flink, Spark, Dataflow, etc.). 💡 Why use Flink Runner with Beam? Let’s dive in! 🏊‍♂️ 👇 🧵[2/X] #DataEngineering #ApacheFlink #ApacheBeam #ETL #DataScience #Tech #Programming

thecodemancer_'s tweet image. Apache Beam lets you write one pipeline & execute it on multiple engines (Flink, Spark, Dataflow, etc.). 💡

Why use Flink Runner with Beam? Let’s dive in! 🏊‍♂️

👇

🧵[2/X]

#DataEngineering #ApacheFlink #ApacheBeam #ETL #DataScience #Tech #Programming

🌐🗺️ Discover the power of Google Cloud Platform (GCP) for GIS workloads! Dong Sun will introduce Shapefile and TIFF formats, discuss loading into BigQuery, and share best practices. Unlock the potential of cloud-based GIS with GCP! 🌍💻 beamsummit.org/program/ #ApacheBeam

ApacheBeam's tweet image. 🌐🗺️ Discover the power of Google Cloud Platform (GCP) for GIS workloads! Dong Sun will introduce Shapefile and TIFF formats, discuss loading into BigQuery, and share best practices. Unlock the potential of cloud-based GIS with GCP! 🌍💻
beamsummit.org/program/
 #ApacheBeam

💡 Here’s something not everyone knows: You can actually run Apache Beam pipelines on Apache Spark as a runner! But that’s a topic for another thread… 👀 🧵[8/X] #DataEngineering #ApacheBeam #Programming #python

thecodemancer_'s tweet image. 💡 Here’s something not everyone knows: You can actually run Apache Beam pipelines on Apache Spark as a runner! But that’s a topic for another thread… 👀

🧵[8/X]

#DataEngineering #ApacheBeam #Programming #python

✅ Fully Managed & Serverless No clusters to manage. No manual scaling. Dataflow auto-scales dynamically, handling resource allocation for you. 🧵[3/X] #DataEngineering #ApacheBeam #Programming #python

thecodemancer_'s tweet image. ✅ Fully Managed & Serverless
No clusters to manage. No manual scaling. Dataflow auto-scales dynamically, handling resource allocation for you.

🧵[3/X]

#DataEngineering #ApacheBeam #Programming #python

✅ One Framework for Batch & Streaming No need to maintain separate batch and streaming jobs. Apache Beam lets you write one pipeline that works for both. 🧵[4/X] #DataEngineering #ApacheBeam #Programming #python

thecodemancer_'s tweet image. ✅ One Framework for Batch & Streaming
No need to maintain separate batch and streaming jobs. Apache Beam lets you write one pipeline that works for both.

🧵[4/X]

#DataEngineering #ApacheBeam #Programming #python

TL;DR: Apache Beam + Flink Runner = flexibility + power. 🔹 Write once, run anywhere 🔹 Multi-language support 🔹 Best-in-class stream & batch processing Curious? Check out Beam’s docs & give it a try! 🚀 🧵[10/X] #DataEngineering #ApacheFlink #ApacheBeam #ETL #DataScience

thecodemancer_'s tweet image. TL;DR: Apache Beam + Flink Runner = flexibility + power.

🔹 Write once, run anywhere
🔹 Multi-language support
🔹 Best-in-class stream & batch processing

Curious? Check out Beam’s docs & give it a try! 🚀

🧵[10/X]

#DataEngineering #ApacheFlink #ApacheBeam #ETL #DataScience

Want to get started? 🏁 1️⃣ Install Apache Beam 2️⃣ Choose Flink Runner 3️⃣ Deploy on Flink (local, Yarn, Kubernetes, etc.) Pre-built Flink Job Service Docker images are available for easy deployment! 🐳 🧵[9/X] #DataEngineering #ApacheFlink #ApacheBeam #ETL #DataScience #Tech

thecodemancer_'s tweet image. Want to get started? 🏁

1️⃣ Install Apache Beam
2️⃣ Choose Flink Runner
3️⃣ Deploy on Flink (local, Yarn, Kubernetes, etc.)

Pre-built Flink Job Service Docker images are available for easy deployment! 🐳

🧵[9/X]

#DataEngineering #ApacheFlink #ApacheBeam #ETL #DataScience #Tech

🤔 So why isn’t Dataflow more widely adopted? Spark has been around longer and is the default for many teams. Also, Beam’s model is different, and that can take some adjustment. 🧵[6/X] #DataEngineering #ApacheBeam #Programming #python

thecodemancer_'s tweet image. 🤔 So why isn’t Dataflow more widely adopted?

Spark has been around longer and is the default for many teams. Also, Beam’s model is different, and that can take some adjustment.

🧵[6/X]

#DataEngineering #ApacheBeam #Programming #python

Apache Flink is the best choice for streaming, right? 🚀 But what if I told you there’s a way to make it even more flexible? 👇 🧵[1/X] #DataEngineering #ApacheFlink #ApacheBeam #ETL #DataScience #Tech #Programming

thecodemancer_'s tweet image. Apache Flink is the best choice for streaming, right? 🚀 But what if I told you there’s a way to make it even more flexible?

👇

🧵[1/X]  

#DataEngineering #ApacheFlink #ApacheBeam #ETL #DataScience #Tech #Programming

Loading...

Something went wrong.


Something went wrong.


United States Trends