Back to library
Data Science

Pandas Data Pipeline & Vectorization Optimizer

Refactors slow, memory-hogging Pandas dataframes using vectorization, categorical types, and efficient memory layouts.

pandaspythonvectorizationdata-wranglingperformance

Prompt

Act as a Principal Data Engineer and Python Performance Specialist.

Optimize and clean the following Pandas data processing code / pipeline:
[Paste your Pandas code and describe dataset schema and row count]

Execute this data engineering optimization:
1. Memory Footprint Reduction:
   - Convert `object` columns to `category` where cardinality is low.
   - Downcast integers (`int64` -> `int32`/`int16`) and floats (`float64` -> `float32`).
   - Parse dates efficiently at load time.
2. Vectorization vs Iteration:
   - Eliminate `for` loops, `.iterrows()`, and `.apply()` in favor of native NumPy vectorization or `eval()`.
3. Defensive Data Cleaning:
   - Handle missing values (`NaN`), duplicate index keys, and outliers systematically.
   - Implement type assertions and schema validation checks.
4. Complete Optimized Code: Provide the revised, clean script with execution speed benchmarks and memory profiling before vs after.

AI Quality Score

95/ 100
Clarity––
Structure––
Token efficiency––
Robustness––
Safety & Guardrails––
Versatility & Adaptability––

Full breakdown across clarity, structure, token efficiency, and robustness — plus improvement suggestions — is available on the paid plan.

Upgrade to unlock