california-housing-api

grader verdict: FAIL (exit 0)
served - ...
FAILED ../../tests/test_outputs.py::test_model_types_trained - FileNotFoundEr...
FAILED ../../tests/test_outputs.py::test_model_prediction_on_hidden_test - Fi...
FAILED ../../tests/test_outputs.py::test_api_prediction_matches_model - FileN...
FAILED ../../tests/test_outputs.py::test_validation_vs_test_metrics - FileNot...
FAILED ../../tests/test_outputs.py::test_predictions_reproducible_from_saved_model
========================= 26 failed, 4 passed in 4.69s =========================

You are an expert terminal agent. You MUST use the provided tools to complete tasks. Use the terminal tool to run shell commands, read_file to read files, write_file to write files, search_files to search, and patch to edit files. Do NOT write out solutions as text - execute them using the tools. Always start by exploring the environment with terminal commands.

# California Housing Price Prediction with FastAPI

## Goal

Build a complete machine learning regression pipeline that predicts California housing prices. Train multiple regression models, select the best performer, deploy it as a FastAPI web service, and validate predictions through API calls.

## Background

You will work with the California Housing dataset. The dataset is pre-split and provided as CSV files:
- `/app/data/train.csv`: Training data with 8 features and target column (16512 samples)
- `/app/data/test.csv`: Test data with 8 features only, no target column (4128 samples)

The 8 features are: MedInc, HouseAge, AveRooms, AveBedrms, Population, AveOccup, Latitude, Longitude
The target is median house value in $100,000s.

Available libraries: scikit-learn (1.6.0), FastAPI (0.115.6), uvicorn (0.34.0), pandas (2.2.3), numpy (2.2.1), requests (2.32.3).

## Steps

### Step 1. Load and Prepare Data

1.1. Unzip the data files:
- Unzip `/app/data/train.zip` to get `/app/data/train.csv`
- Unzip `/app/data/test.zip` to get `/app/data/test.csv`

1.2. Load training data from `/app/data/train.csv`:
- This file contains 8 feature columns and 1 target column (9 columns total)

1.3. Split the training data into train and validation sets:
- Use sklearn's train_test_split
- test_size=0.2
- random_state=42
- Reset DataFrame index before splitting to ensure consistent indexing

1.4. Save split information to `/app/output/split_info.json`:
```json
{
  "train_size": <int>,
  "validation_size": <int>,
  "random_state": 42,
  "test_size": 0.2,
  "train_indices": [<int>, ...],
  "validation_indices": [<int>, ...]
}
```
- train_indices and validation_indices are 0-based row indices from the reset DataFrame
- Indices should be sorted in ascending order

### Step 2. Train Multiple Regression Models

Train at least THREE different regression models:

**Model 1 - Linear Regression:**
- Use LinearRegression from sklearn.linear_model

**Model 2 - Ridge Regression:**
- Use Ridge from sklearn.linear_model

**Model 3 - Random Forest Regressor:**
- Use RandomForestRegressor from sklearn.ensemble

For EACH model:
- Train on the train set (X_train, y_train)
- Calculate training R² score
- Calculate validation R² score
- Calculate validation set metrics: MAE, MSE, RMSE

### Step 3. Select Best Model and Save All Models

3.1. Compare all models based on validation R² score (higher is better)

3.2. Select the model with the highest validation R² score

3.3. Save ALL trained models to `/app/models/` directory:
- `/app/models/linear_regression.pkl`
- `/app/models/ridge.pkl`
- `/app/models/random_forest.pkl`
- `/app/models/best_model.pkl` (copy of the best performing model)
- Save the trained scikit-learn model object directly (not wrapped in a dictionary or metadata object) so that pickle.load() returns an object with a callable predict() method

3.4. Save model comparison to `/app/output/model_comparison.json`:
```json
{
  "models": [
    {
      "model_name": "linear_regression",
      "train_r2": <float>,
      "val_r2": <float>,
      "mae": <float>,
      "mse": <float>,
      "rmse": <float>
    },
    ...
  ],
  "best_model_name": "<string>"
}
```

3.5. Validation quality requirements:
- A properly trained model should achieve on validation set:
  - R² > 0.6
  - MAE < 0.8

### Step 4. Build FastAPI Application

Create a FastAPI application at `/app/api/app.py` with the following requirements:

**Implementation Notes (Recommended):**
- Load the best model from `/app/models/best_model.pkl` at application startup
- Store the model in a module-level or global variable for reuse across requests
- Consider using a startup event handler (e.g., @app.on_event("startup") or lifespan context manager)

**Endpoints:**

**Endpoint 1: POST /predict**
- Input schema:
  ```json
  {
    "features": [float, float, float, float, float, float, float, float]
  }
  ```
- Use the pre-loaded model from application startup
- Output schema:
  ```json
  {
    "prediction": <flo
… (4272 more chars)

I'll build the complete California Housing Price Prediction pipeline with FastAPI. Let me start by exploring the environment and then proceed step by step.

terminal({"command": "ls -la /app/"})

{"output": "total 24\ndrwxr-xr-x 1 root root 4096 Mar  4 09:21 .\ndrwxr-xr-x 1 root root 4096 Apr 21 04:24 ..\ndrwxr-xr-x 2 root root 4096 Mar  4 09:21 api\ndrwxr-xr-x 2 root root 4096 Mar  4 09:21 data\ndrwxr-xr-x 2 root root 4096 Mar  4 09:21 models\ndrwxr-xr-x 2 root root 4096 Mar  4 09:21 output", "exit_code": 0, "error": null}

{"output": "total 736\ndrwxr-xr-x 2 root root   4096 Mar  4 09:21 .\ndrwxr-xr-x 1 root root   4096 Mar  4 09:21 ..\n-rw-rw-r-- 1 root root 139441 Mar  4 09:05 test.zip\n-rw-rw-r-- 1 root root 601889 Mar  4 09:05 train.zip", "exit_code": 0, "error": null}

Let me start by unzipping the data files and exploring the structure:

terminal({"command": "cd /app/data && unzip train.zip && unzip test.zip"})

{"output": "Archive:  train.zip\n  inflating: train.csv               \nArchive:  test.zip\n  inflating: test.csv", "exit_code": 0, "error": null}

{"output": "total 2572\ndrwxr-xr-x 1 root root    4096 Apr 21 04:25 .\ndrwxr-xr-x 1 root root    4096 Mar  4 09:21 ..\n-rwxrwxrwx 1 root root  354101 Dec 22 08:24 test.csv\n-rw-rw-r-- 1 root root  139441 Mar  4 09:05 test.zip\n-rwxrwxrwx 1 root root 1516656 Dec 22 08:24 train.csv\n-rw-rw-r-- 1 root root  601889 Mar  4 09:05 train.zip\nMedInc,HouseAge,AveRooms,AveBedrms,Population,AveOccup,Latitude,Longitude,target\n3.2596,33.0,5.017656500802568,1.0064205457463884,2300.0,3.691813804173355,32.71,-117.03,1.03\n3.8125,49.0,4.473544973544974,1.041005291005291,1314.0,1.7380952380952381,33.77,-118.16,3.821\n4.1563,4.0,5.645833333333333,0.9851190476190477,915.0,2.7232142857142856,34.66,-120.48,1.726\n1.9425,36.0,4.002816901408451,1.0338028169014084,1418.0,3.9943661971830986,32.69,-117.11,0.934", "exit_code": 0, "error": null}

Now let me start implementing the pipeline. First, I'll create the Python script for data loading and model training:

<tool_call>
{"name": "write_file", "arguments": {"path": "/app/pipeline.py", "content": "import pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LinearRegression, Ridge\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error\nimport json\nimport pickle\nimport os\n\n# Step 1: Load and Prepare Data\nprint(\"Step 1: Loading and preparing data...\")\n\ntrain_df = pd.read_csv('/app/data/train.csv')\n\n# Reset index to get consistent indices\ntrain_df.reset_index(drop=True, inplace=True)\n\n# Split into features and target\nX = train_df.drop('target', axis=1)\ny = train_df['target']\n\n# Split into train and validation sets\nX_train, X_val, y_train, y_val, train_indices, val_indices = train_test_split(\n    X, y, train_df.index, \n    test_size=0.2, \n    random_state=42\n)\n\n# Save split information\nsplit_info = {\n    \"train_size\": len(X_train),\n    \"validation_size\": len(X_val),\n    \"random_state\": 42,\n    \"test_size\": 0.2,\n    \"train_indices\": sorted(train_indices),\n    \"validation_indices\": sorted(val_indices)\n}\n\nos.makedirs('/app/output', exist_ok=True)\nwith open('/app/output/split_info.json', 'w') as f:\n    json.dump(split_info, f, indent=2)\n\nprint(f\"Training set size: {len(X_train)}\")\nprint(f\"Validation set size: {len(X_val)}\")\n\n# Step 2: Train Multiple Regression Models\nprint(\"\\nStep 2: Training regression models...\")\n\nmodels = {\n    \"linear_regression\": LinearRegression(),\n    \"ridge\": Ridge(),\n    \"random_forest\": RandomForestRegressor(n_estimators=100, random_state=42)\n}\n\nmodel_metrics = []\n\ntrained_models = {}\n\nfor name, model in models.items():\n    print(f\"Training {name}...\")\n    \n    # Train model\n    model.fit(X_train, y_train)\n    trained_models[name] = model\n    \n    # Predict on train set\n    y_train_pred = model.predict(X_train)\n    train_r2 = r2_score(y_train, y_train_pred)\n    \n    # Predict on validation set\n    y_val_pred = model.predict(X_val)\n    val_r2 = r2_score(y_val, y_val_pred)\n    mae = mean_absolute_error(y_val, y_val_pred)\n    mse = mean_squared_error(y_val, y_val_pred)\n    rmse = np.sqrt(mse)\n    \n    # Store metrics\n    metrics = {\n        \"model_name\": name,\n        \"train_r2\": train_r2,\n        \"val_r2\": val_r2,\n        \"mae\": mae,\n        \"mse\": mse,\n        \"rmse\": rmse\n    }\n    model_metrics.append(metrics)\n    \n    print(f\"{name} - Train R\u00b2: {train_r2:.4f}, Val R\u00b2: {val_r2:.4f}, MAE: {mae:.4f}\")\n\n# Step 3: Select Best Model and Save All Models\nprint(\"\\nStep 3: Selecting and saving models...\")\n\n# Find best model based on validation R\u00b2\nbest_model_name = max(model_metrics, key=lambda x: x[\"val_r2\"])[\"model_name\"]\n\n# Save all models\nos.makedirs('/app/models', exist_ok=True)\nfor name, model in trained_models.items():\n    with open(f'/app/models/{name}.pkl', 'wb') as f:\n        pickle.dump(model, f)\n\n# Save best model\nwith open('/app/models/best_model.pkl', 'wb') as f:\n    pickle.dump(trained_models[best_model_name], f)\n\n# Save model comparison\nmodel_comparison = {\n    \"models\": model_metrics,\n    \"best_model_name\": best_model_name\n}\n\nwith open('/app/output/model_comparison.json', 'w') as f:\n    json.dump(model_comparison, f, indent=2)\n\nprint(f\"Best model: {best_model_name}\")\n\n# Step 4: Build FastAPI Application\nprint(\"\\nStep 4: Creating FastAPI application...\")\n\nfastapi_code = \"\"\"\nimport pickle\nfrom fastapi import FastAPI, HTTPException\nfrom pydantic import BaseModel\nimport numpy as np\nimport os\n\napp = FastAPI()\n\n# Load model at startup\nmodel = None\nmodel_name = None\nmodel_type = None\n\n@app.on_event(\"startup\")\nasync def load_model():\n    global model, 
… (2880 more chars)