Comparing machine learning algorithms
Problem description
A systematic comparison of six classification algorithms on the real Wine data with the same split and metrics, and an understanding of how normalization and the default settings affect the result.
Objectives
- Designing a fair comparison (the same folds and metrics)
- Reading the standard deviation, not just the mean
- Understanding the effect of normalization
Problem type
Classification
Dataset
Tools used
Algorithms
Implementation steps
- In KNIME: File → Import KNIME Workflow… and choose algorithms-comparison.knwf, then Execute all.
- Open algorithms-comparison.ows in Orange: six algorithms connected to a single Test and Score.
- In WEKA: Experimenter → add wine.arff and six algorithms → Run → Analyse with Paired T-Tester.
- Use the Academy's model comparison lab for a quick experiment in the browser.
Workflow diagram
Orange
DataEvaluateModelVisualize
↔ Drag the diagram sideways to see all of it.
Connections as text (10)
- File → Data Table (Data)
- File → Test and Score (Data)
- Tree → Test and Score (Learner)
- Random Forest → Test and Score (Learner)
- kNN → Test and Score (Learner)
- SVM → Test and Score (Learner)
- Logistic Regression → Test and Score (Learner)
- Naive Bayes → Test and Score (Learner)
- Test and Score → Confusion Matrix (Evaluation Results)
- File → Scatter Plot (Data)
KNIME
SourceManipulatorLearnerPredictorOtherData portModel port
↔ Drag the diagram sideways to see all of it.
Connections as text (11)
- CSV Reader → Partitioning
- Partitioning → Decision Tree Learner
- Decision Tree Learner → Decision Tree Predictor
- Decision Tree Predictor → Scorer
- Partitioning → Decision Tree Predictor
- Partitioning → Random Forest Learner
- Random Forest Learner → Random Forest Predictor
- Random Forest Predictor → Scorer
- Partitioning → Random Forest Predictor
- Partitioning → K Nearest Neighbor
- K Nearest Neighbor → Scorer
Project files
- CSV datasetCSV13.0 KB
- WEKA ARFF fileWEKA ARFF13.8 KB
- Orange workflow (.ows)Orange8.2 KB
- Data in Orange format (.tab) for the workflowOrange TAB13.2 KB
- Computed reference results (CSV)Results320.0 B
- KNIME workflow (.knwf): ran successfully in the KNIME 5.12 engine (not opened in the interface)KNIME65.9 KB
- Step-by-step implementation guide (PDF)PDF684.2 KB
- Full project package (ZIP)ZIP690.7 KB
The KNIME file (.knwf) was built on KNIME 5.12 and all its nodes ran successfully in the KNIME engine without opening the interface; open it from File → Import KNIME Workflow.
Results
Actual results computed on the attached files
• Decision Tree: Accuracy 90.0% (± 7.8%), F1-macro 0.902
• Random Forest: Accuracy 98.9% (± 2.2%), F1-macro 0.989
• kNN (k=5): Accuracy 97.2% (± 4.5%), F1-macro 0.973
• SVM (RBF): Accuracy 97.8% (± 3.7%), F1-macro 0.977
• Logistic Regression: Accuracy 98.3% (± 2.5%), F1-macro 0.983
• Naive Bayes: Accuracy 97.8% (± 3.7%), F1-macro 0.978
The scikit-learn reference normalizes the features before kNN, SVM and Logistic Regression.
And the results of the Orange 3.41.0 library (10-fold, like the default Test and Score settings):
• Tree: CA 91.0%, F1 0.910, AUC 0.931
• Random Forest: CA 97.2%, F1 0.972, AUC 0.997
• kNN: CA 67.4%, F1 0.667, AUC 0.865
• SVM: CA 98.3%, F1 0.983, AUC 0.987
• Logistic Regression: CA 94.4%, F1 0.944, AUC 0.992
• Naive Bayes: CA 97.2%, F1 0.972, AUC 0.997
Evaluation metrics
- Accuracy with the standard deviation
- F1-macro
- ROC AUC
Interpreting the results
Most algorithms exceed 97% in the scikit-learn reference, and the differences between them are within the standard deviation, so it is not right to declare a “winner” without a statistical test. Note the drop of kNN in Orange to 67.4%, because the distances here are computed on non-normalized features (proline, in the hundreds, dominates the others); add Preprocess → Normalize before kNN and repeat the experiment. We verified with the Orange library itself that adding normalization raises kNN to 96.1% with the same folds.
Discussion questions
- Why don't we pick a winner from a 0.5% difference?
- What is the effect of normalization on kNN, SVM and trees?
- How would you design a comparison in which test information does not leak into the normalization?
Try it in the Model Comparison Lab
