Comparing machine learning algorithms

AdvancedClassification

Problem description

A systematic comparison of six classification algorithms on the real Wine data with the same split and metrics, and an understanding of how normalization and the default settings affect the result.

Objectives

  • Designing a fair comparison (the same folds and metrics)
  • Reading the standard deviation, not just the mean
  • Understanding the effect of normalization

Problem type

Classification

Dataset

Tools used

Algorithms

Implementation steps

  1. In KNIME: File → Import KNIME Workflow… and choose algorithms-comparison.knwf, then Execute all.
  2. Open algorithms-comparison.ows in Orange: six algorithms connected to a single Test and Score.
  3. In WEKA: Experimenter → add wine.arff and six algorithms → Run → Analyse with Paired T-Tester.
  4. Use the Academy's model comparison lab for a quick experiment in the browser.

Workflow diagram

Orange

DataEvaluateModelVisualize

File FileData Table Data TableTest and Score Test and ScoreTree TreeRandom Forest Random ForestkNN kNNSVM SVMLogistic Regression LogisticRegressionNaive Bayes Naive BayesConfusion Matrix Confusion MatrixScatter Plot Scatter PlotDataDataLearnerLearnerLearnerLearnerLearnerLearnerEvaluation ResultsData
Drawn from the OWS workflow file attached to the project: the widgets, their icons, their category colours and the channel names on the links, as they appear when opened in Orange.

↔ Drag the diagram sideways to see all of it.

Connections as text (10)
  • File → Data Table (Data)
  • File → Test and Score (Data)
  • Tree → Test and Score (Learner)
  • Random Forest → Test and Score (Learner)
  • kNN → Test and Score (Learner)
  • SVM → Test and Score (Learner)
  • Logistic Regression → Test and Score (Learner)
  • Naive Bayes → Test and Score (Learner)
  • Test and Score → Confusion Matrix (Evaluation Results)
  • File → Scatter Plot (Data)

KNIME

SourceManipulatorLearnerPredictorOtherData portModel port

CSV Reader — SourceCSV ReaderNode 1Partitioning — ManipulatorPartitioningNode 2Decision Tree Learner — LearnerDecision Tree LearnerNode 3Decision Tree Predictor — PredictorDecision TreePredictorNode 4Scorer — OtherScorerNode 5Random Forest Learner — LearnerRandom Forest LearnerNode 6Random Forest Predictor — PredictorRandom ForestPredictorNode 7Scorer — OtherScorerNode 8K Nearest Neighbor — LearnerK Nearest NeighborNode 9Scorer — OtherScorerNode 10
KNIME nodes with their names in the node repository, their original icons, type colour, shape and ports (a triangle for data, a square for a model) and the status light, so you can build it in the same order. The ready KNWF workflow file is attached under "Project files", and it ran successfully in the KNIME 5.12 engine.

↔ Drag the diagram sideways to see all of it.

Connections as text (11)
  • CSV Reader → Partitioning
  • Partitioning → Decision Tree Learner
  • Decision Tree Learner → Decision Tree Predictor
  • Decision Tree Predictor → Scorer
  • Partitioning → Decision Tree Predictor
  • Partitioning → Random Forest Learner
  • Random Forest Learner → Random Forest Predictor
  • Random Forest Predictor → Scorer
  • Partitioning → Random Forest Predictor
  • Partitioning → K Nearest Neighbor
  • K Nearest Neighbor → Scorer

Project files

The KNIME file (.knwf) was built on KNIME 5.12 and all its nodes ran successfully in the KNIME engine without opening the interface; open it from File → Import KNIME Workflow.

Results

Actual results computed on the attached files

scikit-learn 1.9.1 reference, 10-fold stratified CV, shuffle, random_state=42:
• Decision Tree: Accuracy 90.0% (± 7.8%), F1-macro 0.902
• Random Forest: Accuracy 98.9% (± 2.2%), F1-macro 0.989
• kNN (k=5): Accuracy 97.2% (± 4.5%), F1-macro 0.973
• SVM (RBF): Accuracy 97.8% (± 3.7%), F1-macro 0.977
• Logistic Regression: Accuracy 98.3% (± 2.5%), F1-macro 0.983
• Naive Bayes: Accuracy 97.8% (± 3.7%), F1-macro 0.978
The scikit-learn reference normalizes the features before kNN, SVM and Logistic Regression.
And the results of the Orange 3.41.0 library (10-fold, like the default Test and Score settings):
• Tree: CA 91.0%, F1 0.910, AUC 0.931
• Random Forest: CA 97.2%, F1 0.972, AUC 0.997
• kNN: CA 67.4%, F1 0.667, AUC 0.865
• SVM: CA 98.3%, F1 0.983, AUC 0.987
• Logistic Regression: CA 94.4%, F1 0.944, AUC 0.992
• Naive Bayes: CA 97.2%, F1 0.972, AUC 0.997

Evaluation metrics

  • Accuracy with the standard deviation
  • F1-macro
  • ROC AUC

Interpreting the results

Most algorithms exceed 97% in the scikit-learn reference, and the differences between them are within the standard deviation, so it is not right to declare a “winner” without a statistical test. Note the drop of kNN in Orange to 67.4%, because the distances here are computed on non-normalized features (proline, in the hundreds, dominates the others); add Preprocess → Normalize before kNN and repeat the experiment. We verified with the Orange library itself that adding normalization raises kNN to 96.1% with the same folds.

Discussion questions

  • Why don't we pick a winner from a 0.5% difference?
  • What is the effect of normalization on kNN, SVM and trees?
  • How would you design a comparison in which test information does not leak into the normalization?

Try it in the Model Comparison Lab

Similar projects