Step-by-Step Tutorial

In this tutorial you will use MANILA to perform a fairness and effectiveness evaluation of different machine learning settings (i.e., machile learning model and fairness-enhancing methods) to predict recidivism of condemned people using the COMPAS dataset.

In particular, you will evaluate the fairness and effectiveness of the following settings using the following metrics:

• Settings: Logistic Regression, and Random Forest with and without the following fairness-enhancing methods: Reweighing, and Debiaser for Multiple Variables (DEMV).

• Metrics: Accuracy, Disparate Impact, Equalized Odds.

Finally, you will use the Harmonic Mean as aggregation function to obtain a single score for each setting.

MANILA will independently evaluate each setting using the selected metrics and identify the best setting according to the selected aggregation function.

In the following, we show the steps to perform the evaluation.

• 1. Obtain the dataset
• 2. Access MANILA
  • From the SoBigData RI
  • Local installation
• 3. Use MANILA
  • 3.1. Dataset
  • 3.2. Scaler
  • 3.3. ML Task
  • 3.4. Fairness Methods
  • 3.5. Metrics
  • 3.6. Validation
  • 3.7. Upload dataset
  • 3.8. Run the experiment
• 4. Visualizing the Results