Posts about Apache Spark
I am checking stack overflow survey 2022 (https://survey.stackoverflow.co/2022/#top-paying-technologies-other-frameworks-and-libraries), and I see that apache spark is the highest paying framework under other frameworks category. I want to upskill myself and to be more demanded in job market. So is it worth learning Apache Spark (PySpark) in 2023?
The awesome-spark repo has a list of Spark OSS libraries, but a lot of them are quite old.
I am thinking about curating another list that's a bit more focused and updated. For example, I'm interested in knowing all the good data validation libraries that are currently being maintained for PySpark right now. It's hard to figure out the best options.
Feel free to add the libraries you like a lot in the comments and I'll try to collate a list.