Feature Engineering and Model Inputs

Before raw data reaches a machine learning model, it must be transformed into a format the model can process. This transformation — feature engineering in classical ML, tokenization and embedding in modern deep learning — fundamentally shapes what the model can perceive and therefore what it can learn. Understanding inputs helps you reason about model blind spots.

What Are Features?

In traditional machine learning, features are measurable properties extracted from raw data. For a house price predictor: square footage, number of bedrooms, neighborhood crime rate. Engineers manually select and craft features they believe are predictive.

Good features make patterns easy to learn. Bad features — irrelevant, redundant, or misleading ones — confuse the model or introduce bias.

Deep Learning Changed the Game

Neural networks, especially large language models, largely automate feature learning. Instead of hand-crafting features, deep models learn hierarchical representations directly from raw inputs. The first layers of an image network learn edges; deeper layers learn shapes, objects, and scenes. LLMs learn token-level patterns that build into syntactic, semantic, and factual representations.

However, the input preprocessing still matters enormously:

Tokenization splits text into subword units. Unusual words, code, non-English text, and emoji may be split unexpectedly, affecting comprehension.
Context windows limit how much text the model sees at once. Information beyond the window is invisible.
Prompt formatting — system messages, delimiters, instruction templates — shapes how the model interprets the task.

Implications for Evaluation

The model only sees what you give it. If a task provides incomplete context, poor formatting, or ambiguous instructions, distinguish between model failure and input failure. A fair evaluation accounts for what information was actually available.

Unusual inputs expose weaknesses. Rare languages, specialized jargon, mathematical notation, and mixed-format content stress the preprocessing pipeline. These are high-value test cases.

Compare apples to apples. When ranking model outputs, ensure each model received the same input format. Differences in prompting can explain quality gaps that look like model differences.

Practical Evaluation Checklist

Is the prompt clear and complete?
Does the task require information beyond the model's context window?
Are there formatting quirks that might confuse the model?
Would a human expert with the same input produce a good answer?

Key Takeaways

Models can only learn from the inputs they receive, after preprocessing
Tokenization, context limits, and prompt format all affect output quality
Distinguish model failures from input or context failures in your evaluations
Unusual inputs are valuable stress tests for preprocessing pipelines