This project demonstrates data engineering practices for computer vision. The pipeline securely extracts CIFAR-10, transforms raw batch files into model-ready RGB tensors, creates reproducible train and validation splits, and generates visual checks to validate data quality before training.
Highlights Secure archive extraction with path traversal protection. Reproducible train and validation splitting with class-balance preservation. Clean, modular workflow organized through reusable functions. Visual verification output for rapid data quality checks. Automatic class-name mapping from metadata with fallback handling. Dataset Overview The CIFAR-10 dataset contains 60,000 color images (32x32) in 10 classes.
Training images: 50,000 (split to train and validation) Test images: 10,000 Image tensor shape: (N, 32, 32, 3) Class mapping:
Label Class 0 Airplane 1 Automobile 2 Bird 3 Cat 4 Deer 5 Dog 6 Frog 7 Horse 8 Ship 9 Truck Requirements Python 3.9+ NumPy scikit-learn Matplotlib Install dependencies:
pip install -r requirements.txt Setup Download CIFAR-10 from the official source: https://www.cs.toronto.edu/~kriz/cifar.html Place the downloaded file named cifar-10-python.tar.gz in the project root. Keep the script and archive in the same folder, or pass a custom archive path. Usage Default run:
python cifar10_preprocessing.py If the python command is not configured on Windows, use:
py -3 cifar10_preprocessing.py Headless run for terminal-only or automation environments:
python cifar10_preprocessing.py --no-plot Custom archive and extraction paths:
python cifar10_preprocessing.py --archive ./cifar-10-python.tar.gz --extract-dir . CLI Options --archive: Path to the CIFAR-10 tar.gz file. --extract-dir: Directory where dataset files should be extracted. --validation-size: Fraction of training data reserved for validation (default: 0.2). --random-state: Seed for reproducible splitting (default: 42). --no-plot: Skip the interactive preview window. --save-preview: Output path for the preview image (default: result_preview.png). Expected Output When the script runs, it prints:
Train, validation, and test array shapes. Unique class labels found in the training split. The label numbering convention (0 through 9). A True/False check confirming labels 0 through 9 are present. The saved preview image location. A preview image is saved to result_preview.png by default.
Preview The generated preview image is included in this repository:
CIFAR-10 preview
Repository Structure . |-- cifar10_preprocessing.py |-- README.md |-- requirements.txt |-- result_preview.png |-- .gitignore `-- .gitattributes Project Capabilities Included:
Archive extraction and binary batch loading. Data reshaping and train or validation splitting. Data verification and sample visualization. Current boundaries:
Model training or evaluation. Data augmentation pipeline. Experiment tracking.