The Role of Model Selection

This project started with a practical classification problem: given OCR text extracted from scanned documents, can a machine learning model identify whether a document belongs to a specific target document class?

The data came from OCR outputs, which means the text was not clean. It contained scanner artifacts, broken words, page headers, fax metadata, timestamps, provider names, repeated fragments, inconsistent spacing, and formatting noise. The goal was not simply to build a model that performed well on one train/test split. The real goal was to build a model that could generalize across newly collected OCR data.

The modeling process evolved in six stages:

1. Fine-tuning a BERT-based classifier.
2. Building a TF-IDF + Linear SVM model.
3. Finding and fixing a major data leakage issue caused by empty text values.
4. Validating the SVM on new external OCR data.
5. Implementing Pegasos-style SVM training to understand stochastic SVM optimization.
6. Comparing production SVM performance against manual Pegasos experiments.

The most important lesson from the project was that model accuracy alone is not enough. Data quality, leakage detection, external validation, and error analysis were just as important as model selection.

The starting point: OCR text classification

The classification task was binary.

The input was OCR text. Each row represented one document. The model learned from text features and predicted whether the document belonged to the target class.

At a high level, the pipeline looked like this:

Because OCR text can be long and messy, I tested both deep learning and classical machine learning approaches.

Stage 1: fine-tuning BERT

The first modeling approach used BERT for sequence classification. The text was tokenized using a BERT tokenizer, then passed into a BERT classification head with two output labels.

The first BERT model used a maximum token length of 256. This means the model only saw the first 256 tokens of each OCR document. Training used weighted cross-entropy loss to address class balance.

The BERT 256-token model performed strongly.

This was already strong performance. However, error analysis showed an important limitation: some false negatives were not borderline cases. The model was very confident they were non-target documents even though they were target documents.

That led to an important observation: the evidence needed for classification sometimes appeared after the first 256 tokens. Since the model only saw the beginning of the OCR text, it could miss the most useful parts of the document.

To test this, I retrained BERT with a maximum token length of 512.

The 512-token model performed better than the 256-token version, but the core limitation remained: BERT still has a fixed input length. Long OCR documents can contain important signals beyond the model’s visible window.

This pushed the project toward classical text classification.

Stage 2: moving to TF-IDF + Linear SVM

The next model used TF-IDF features with a Linear SVM. This approach is much simpler than BERT, but it has several practical advantages for OCR classification:

• It can use the full OCR text rather than only the first 256 or 512 tokens.
• It works very well with sparse text signals.
• It is fast to train and easy to inspect.
• It is strong for document classification when classes have repeated phrases, templates, or domain-specific vocabulary.

The vectorizer used word n-grams from one to three words, sublinear term frequency, and a large feature space.

The classifier used stochastic hinge-loss optimization.

This is effectively a large-scale linear SVM trained with stochastic optimization. The loss="hinge" setting makes it an SVM-style classifier, while penalty="l2" controls the weight size through regularization.

The regularization parameter was alpha = 1e-5. Conceptually, this plays a similar role to the lambda parameter in Pegasos-style SVM optimization. A smaller value allows the model to fit the data more closely, while a larger value forces stronger regularization.

The first SVM experiment looked extremely strong, but that result turned out to be misleading.

The data leakage discovery

At one point, the SVM produced an almost perfect result with only one error. That looked impressive, but it was suspicious.

A deeper diagnostic check revealed the issue: many rows in the non-target class had empty text in the column being used for training.

The duplicate hash check exposed the problem. The MD5 hash for the empty string appeared repeatedly:

This hash corresponds to an empty string. The analysis showed that a very large number of non-target rows had empty values in the text column, while the actual OCR text existed in another column.

This created a data leakage problem. The model could learn a dataset artifact instead of the real document signal:

That is not real classification. It is learning a shortcut. The inflated result had to be rejected.

This was one of the most important lessons in the project: a model can achieve excellent metrics for the wrong reason.

Fixing the dataset

The dataset was repaired by checking both available text columns. If the main text column was empty, the pipeline used the backup OCR text column.

The repair logic was:

After fixing the text field, the dataset looked much more valid.

After deduplication, the corrected dataset contained:

A conflict check was also performed to make sure the same exact OCR text did not appear with both labels. No conflicting duplicate labels were found.

This step made the later model results much more defensible.

Stage 3: SVM on the corrected dataset

After fixing the empty-text issue and removing exact duplicates, I trained the TF-IDF + Linear SVM again.

The confusion matrix was:

This result was still extremely strong, but unlike the earlier inflated result, it was now based on a repaired and deduplicated dataset.

Stage 4: external positive-class recall test

After the corrected SVM performed well internally, the next question was whether it could generalize to newly collected OCR data.

A separate dataset containing 34,394 new target-class OCR records was prepared. This dataset had a nested OCR structure where text was stored inside a pages field. The cleaning pipeline extracted page-level text and combined it into document-level text.

The preparation results were:

This dataset contained only the positive class, so it could not be used to measure full accuracy or precision. Instead, it was used as an external recall test.

The existing SVM trained on the corrected 100K dataset was tested on the 34,303 new positive documents.

This was an important validation step. The model had never trained on this new dataset, yet it detected 99.80% of the target documents.

This suggested that the SVM was not simply memorizing the corrected training set. It generalized well to newly collected OCR data.

Stage 5: building the combined 138K dataset

After the external test, the new 34,303 positive examples were merged into the corrected 100K dataset.

The final combined dataset contained:

Because the combined dataset became more positive-heavy, the SVM retained class_weight="balanced". This prevented the model from over-favoring the larger target class.

The combined dataset was shuffled and split into train/test sets:

The SVM trained on the combined 138K dataset achieved:

The confusion matrix was:

This became the strongest production model result.

Why the SVM worked so well

The SVM performed extremely well because this classification problem had strong lexical and structural signals.

OCR documents often contain repeated domain-specific patterns, such as:

• clinical complaint sections
• history sections
• diagnosis sections
• assessment language
• provider or visit descriptions
• injury and treatment terminology
• repeated document templates
• class-specific phrase patterns

TF-IDF is very effective when the presence, absence, and frequency of terms or phrases carry strong classification information.

The SVM also had an advantage over BERT: it could use the full document text. BERT was limited by token length, while TF-IDF could represent signals appearing anywhere in the OCR text.

Stage 6: testing Pegasos-style SVM

After building the production SVM, I implemented a manual Pegasos-style SVM to better understand how stochastic SVM optimization works.

Pegasos is a primal SVM optimization method that uses stochastic sub-gradient updates. The core objective is:

$\min_{\mathbf{w}} \frac{\lambda}{2} \lVert\mathbf{w}\rVert^2 + \frac{1}{n} \sum_{i=1}^{n} \max(0, 1 - y_i f(x_i))$

The hinge-loss condition is based on the margin:

$\text{margin} = y f(x) = y(\mathbf{w}^{T}\mathbf{x})$

If the margin satisfies

$\text{margin} \geq 1$

the example is correctly classified with enough margin. If

$\text{margin} < 1$

the example violates the margin and contributes to the update.

The learning rate in Pegasos changes over time:

$\eta_t = \frac{1}{\lambda t}$

In the full Pegasos experiment, the settings were:

The learning rate starts large and decreases as training progresses. Early in training, the model makes larger corrections. Later, as t increases, the learning rate becomes smaller, allowing the model to stabilize.

Pegasos on a 30K sample

The first Pegasos experiment used a balanced 30,000-document sample. This was useful for learning and debugging.

The test set contained 6,000 documents.

The most useful training signal was the decline in margin violations:

This showed that Pegasos was learning. By the final epoch, very few examples were violating the margin.

Pegasos on the full 138K dataset

After confirming the algorithm worked on a sample, I trained Pegasos on the full combined dataset.

The confusion matrix was:

The margin violations dropped sharply:

This confirmed that the Pegasos-style optimizer was working well.

However, compared with the production SVM, Pegasos had more false negatives:

If the priority is to avoid missing target documents, the production SVM remains better.

Pegasos on a separate external dataset

The full Pegasos model was also tested on a separate external dataset with 19,286 documents.

The result was:

The confusion matrix was:

This external performance was lower than the internal test performance. That is expected when the new dataset has different OCR quality, formatting, templates, or document distributions.

This showed the value of external validation. Internal test performance can be very high, but external data reveals how much the model depends on the original data distribution.

Model comparison summary

Key observations

1. BERT worked well, but token length was a limitation

BERT performed strongly, but OCR documents are often long. When useful evidence appeared after the token limit, the model could miss it. Increasing the maximum length from 256 to 512 improved results, but did not fully remove the limitation.

2. TF-IDF + Linear SVM was surprisingly strong

The classical SVM outperformed BERT in this case. This happened because the task had strong document-level lexical signals, and TF-IDF could represent the full OCR text rather than only the beginning of the document.

3. Data leakage almost produced a misleading conclusion

The most important data issue was the discovery that many non-target rows had empty text in the modeling column. If this had not been detected, the model would have appeared almost perfect for the wrong reason.

The fix required:

• checking text length by label
• finding repeated empty-string hashes
• repairing empty text using the correct OCR text column
• removing exact duplicates
• checking for conflicting labels

This turned the experiment from a misleading result into a defensible pipeline.

4. External validation was essential

The external positive-only recall test showed that the production SVM could generalize to newly collected OCR data. This was stronger evidence than internal accuracy alone.

5. Pegasos was useful for learning optimization

The manual Pegasos implementation helped explain how SVM optimization works:

• hinge loss
• margin violations
• L2 regularization
• lambda
• decaying learning rate
• stochastic updates
• projection step

Pegasos performed well, but the production SVM remained stronger.

6. External datasets revealed domain shift

The full Pegasos model performed very well internally but dropped on a different external dataset. This suggests that OCR quality, provider templates, document formatting, or labeling differences matter.

Final production choice

The best production model was the TF-IDF + Linear SVM trained on the combined 138K dataset.

Its strengths were:

• highest overall accuracy
• lowest total error count
• very low false negatives
• fast training time
• full-document OCR representation
• strong external recall performance
• easy reproducibility

The final production result was:

For this task, the classical SVM was not just a baseline. It became the strongest model.

Conclusion

This project started with a modern deep learning model and ended with a classical linear model as the best practical solution.

BERT was useful and performed well, but the TF-IDF + Linear SVM better matched the structure of the problem. OCR documents contained repeated terms, templates, and document-level signals that sparse text features could capture very effectively.

The most important part of the project was not simply comparing algorithms. It was the process of making the data trustworthy. The discovery of empty text values in one class showed how easily a model can exploit artifacts instead of learning the real task.

After repairing the dataset, removing duplicates, validating externally, and comparing SVM with Pegasos-style optimization, the final conclusion was clear: a carefully cleaned dataset plus a well-tuned linear SVM produced the most accurate and reliable classifier for this OCR document classification task.

The overall lesson is simple:

In applied machine learning, the best model is not always the most complex model. The best model is the one that learns the real signal, survives external validation, and remains reliable after data quality problems are fixed.