Back to library
Data Science
Pandas Data Pipeline & Vectorization Optimizer
Refactors slow, memory-hogging Pandas dataframes using vectorization, categorical types, and efficient memory layouts.
pandaspythonvectorizationdata-wranglingperformance
Prompt
Act as a Principal Data Engineer and Python Performance Specialist. Optimize and clean the following Pandas data processing code / pipeline: [Paste your Pandas code and describe dataset schema and row count] Execute this data engineering optimization: 1. Memory Footprint Reduction: - Convert `object` columns to `category` where cardinality is low. - Downcast integers (`int64` -> `int32`/`int16`) and floats (`float64` -> `float32`). - Parse dates efficiently at load time. 2. Vectorization vs Iteration: - Eliminate `for` loops, `.iterrows()`, and `.apply()` in favor of native NumPy vectorization or `eval()`. 3. Defensive Data Cleaning: - Handle missing values (`NaN`), duplicate index keys, and outliers systematically. - Implement type assertions and schema validation checks. 4. Complete Optimized Code: Provide the revised, clean script with execution speed benchmarks and memory profiling before vs after.
AI Quality Score
95/ 100
Clarity––
Structure––
Token efficiency––
Robustness––
Safety & Guardrails––
Versatility & Adaptability––
Full breakdown across clarity, structure, token efficiency, and robustness — plus improvement suggestions — is available on the paid plan.
Upgrade to unlock