Skip to main content
Back to registry

spark-optimization

wshobson/agents

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

Installs2
Install command
npx skills add https://github.com/wshobson/agents --skill spark-optimization
Security audits
Gen Agent Trust HubPASS
SocketPASS
SnykPASS
About this skill
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning. - Optimizing slow Spark jobs - Tuning memory and executor configuration - Implementing efficient partitioning strategies - Debugging Spark performance issues - Scaling Spark pipelines for large datasets - Reducing shuffle and data skew - Enable AQE - Adaptive query execution handles many issues - Use Parquet/Delta - Columnar formats with compression - Broadcast small tables - Avoid shuffle for small joins - Monitor Spark UI - Check for skew, spills, GC - Right-size partitions - 128MB - 256MB per partition - Don't collect large data - Keep data distributed - Don't use UDFs unnecessarily - Use built-in functions - Don't over-cache - Memory is limited - Don't ignore data skew - It dominates job time - Don't use .count() for existence - Use .take(1) or .isEmpty()

Source description provided by the upstream skill listing. Community reviews and install context appear in the sections below.

Community Reviews

Latest reviews

Sign in to review

No community reviews yet. Be the first to review.

Browse this skill in context
FAQ
What does spark-optimization do?

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

Is spark-optimization good?

spark-optimization does not have approved reviews yet, so SkillJury cannot publish a community verdict.

What agent does spark-optimization work with?

spark-optimization currently lists compatibility with Agent compatibility has not been published yet..

What are alternatives to spark-optimization?

Skills in the same category include telegram-bot-builder, flutter-app-size, sharp-edges, iterative-retrieval.

How do I install spark-optimization?

npx skills add https://github.com/wshobson/agents --skill spark-optimization

Related skills

More from wshobson/agents

Related skills

Alternatives in Software Engineering