Tracing a prediction: the training-serving boundary

Intuition

Imagine training with attendance recorded as a percentage, 85, while the serving code receives it as a fraction, 0.85. Nothing crashes, and the model returns a neat probability. It is just wrong, for every student, every day. This is the most common kind of production ML bug, because no error message ever appears.

How it works

One request, step by step:

Batch or real-time?

The training-serving boundary is the line between the code that made the model and the code that uses it. Skew means the inputs differ across that line. Common causes:

• two separate implementations of the same feature;
• different units or scales;
• a different meaning for missing values;
• using information at training time that isn't available at serving time.

The main cure is to ship the fitted pipeline with the model, so that the very same code runs on both sides. Topic 22 extends this idea to feature stores.

Run it: trace a request, then cause a skew bug

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.pipeline import make_pipeline
from placement_predict import FEATURES, load_data, preprocessor, train_val_test

df = load_data()
X_train, X_val, X_test, y_train, y_val, y_test = train_val_test(df)
pipeline = preprocessor()
clf = LogisticRegression(max_iter=1000)
model = make_pipeline(pipeline, clf).fit(X_train, y_train)

request = {"CollegeTier": "Tier2", "Stream": "IT", "Specialisation": "DataScience", "Hostel": "No", "HistoryOfBacklogs": "No",
           "CGPA": 8.1, "AttendancePercent": 84, "Internships": 1, "Projects": 3, "Workshops": 2, "Certifications": 2,
           "Publications": 0, "AptitudeTestScore": 70, "SoftSkillsRating": 3.4, "CodingTestScore": 76,
           "MockInterviewScore": 68, "ExtraCurricular": 1}

print("1 Received: ", request["CGPA"], request["Stream"], "...")
assert 0 <= request["CGPA"] <= 10, "CGPA out of range"
print("2 Validated")
row = pd.DataFrame([request]).reindex(columns=FEATURES)
features = pipeline.transform(row)
print("3 Features: ", np.round(features[0][:4], 2), "...", features.shape[1], "values")
score = clf.decision_function(features)[0]
print(f"4 Score:      {score:.3f}")
p = 1 / (1 + np.exp(-score))
print(f"5 Response:   placed_probability = {p:.3f}")

print("\nTraining-serving skew")
print(f"Correct serving code:                accuracy = {accuracy_score(y_val, model.predict(X_val)):.3f}")

skewed = X_val.copy()
skewed["AttendancePercent"] = skewed["AttendancePercent"] / 100   # the serving team used fractions
print(f"Attendance sent as a fraction:       accuracy = {accuracy_score(y_val, model.predict(skewed)):.3f}")

skewed = X_val.copy()
skewed["CGPA"] = skewed["CGPA"] * 9.5                             # the serving team used percentages
print(f"CGPA sent as a percentage:           accuracy = {accuracy_score(y_val, model.predict(skewed)):.3f}")

skewed = X_val.copy()
skewed["MockInterviewScore"] = skewed["MockInterviewScore"].fillna(0)   # a different meaning for missing
print(f"Missing values filled with 0:        accuracy = {accuracy_score(y_val, model.predict(skewed)):.3f}")

print("\nNo error was raised in any of these runs.")

Try this: change the attendance bug to skewed["AttendancePercent"] * 2. Compare what the fractions do with what the larger values do. Which is harder to notice?

Common mistakes

• Rewriting feature code for the serving side. Sooner or later, the two copies will differ.
• Using fit_transform on serving data. That learns new statistics from a single request. Use transform.
• Testing only that the service returns a response. A wrong number looks just like a right one.

Problems

1. In your own words, explain skew to a teammate who has never heard of it.
2. For each, decide on batch or real-time: (a) a weekly report to the placement cell; (b) a chatbot that estimates chances as you type; (c) a yearly retraining job.
3. A feature "offers received so far this year" is used in training. Why is it a problem at prediction time?
4. Add range checks for all numeric fields to the validation step.
5. Write a test that would have caught the fraction bug before release.