← Back to Projects

Computer Vision Preprocessing Pipeline

GitHub ↗

This project demonstrates data engineering practices for computer vision. The pipeline securely extracts CIFAR-10, transforms raw batch files into model-ready RGB tensors, creates reproducible train and validation splits, and generates visual checks to validate data quality before training.

Highlights

  • Secure archive extraction with path traversal protection.
  • Reproducible train and validation splitting with class-balance preservation.
  • Clean, modular workflow organized through reusable functions.
  • Visual verification output for rapid data quality checks.
  • Automatic class-name mapping from metadata with fallback handling.

Dataset Overview

The CIFAR-10 dataset contains 60,000 color images (32x32) in 10 classes.

  • Training images: 50,000 (split to train and validation)
  • Test images: 10,000
  • Image tensor shape: (N, 32, 32, 3)

Class mapping:

| Label | Class | | --- | --- | | 0 | Airplane | | 1 | Automobile | | 2 | Bird | | 3 | Cat | | 4 | Deer | | 5 | Dog | | 6 | Frog | | 7 | Horse | | 8 | Ship | | 9 | Truck |

Requirements

  • Python 3.9+
  • NumPy
  • scikit-learn
  • Matplotlib

Install dependencies:

pip install -r requirements.txt

Setup

  1. Download CIFAR-10 from the official source: https://www.cs.toronto.edu/~kriz/cifar.html
  2. Place the downloaded file named cifar-10-python.tar.gz in the project root.
  3. Keep the script and archive in the same folder, or pass a custom archive path.

Usage

Default run:

python cifar10_preprocessing.py

If the python command is not configured on Windows, use:

py -3 cifar10_preprocessing.py

Headless run for terminal-only or automation environments:

python cifar10_preprocessing.py --no-plot

Custom archive and extraction paths:

python cifar10_preprocessing.py --archive ./cifar-10-python.tar.gz --extract-dir .

CLI Options

  • --archive: Path to the CIFAR-10 tar.gz file.
  • --extract-dir: Directory where dataset files should be extracted.
  • --validation-size: Fraction of training data reserved for validation (default: 0.2).
  • --random-state: Seed for reproducible splitting (default: 42).
  • --no-plot: Skip the interactive preview window.
  • --save-preview: Output path for the preview image (default: result_preview.png).

Expected Output

When the script runs, it prints:

  • Train, validation, and test array shapes.
  • Unique class labels found in the training split.
  • The label numbering convention (0 through 9).
  • A True/False check confirming labels 0 through 9 are present.
  • The saved preview image location.

A preview image is saved to result_preview.png by default.

Repository Structure

.
|-- cifar10_preprocessing.py
|-- README.md
|-- requirements.txt
|-- result_preview.png
|-- .gitignore
`-- .gitattributes

Project Capabilities

Included:

  • Archive extraction and binary batch loading.
  • Data reshaping and train or validation splitting.
  • Data verification and sample visualization.

Current boundaries:

  • Model training or evaluation.
  • Data augmentation pipeline.
  • Experiment tracking.