**Experts' evaluation**

**a) Questionnaire - Weight elicitation**

The FEEM SI 2013 "weights" (fuzzy measures) were derived by means of a specific survey implemented in QUALTRICS web-platform.

This questionnaire is a necessary step to elicit individual preferences in each node and sub-node of the decision tree; we can infer, not only the relative importance of each criteria under consideration but, also the degree of interaction, hence the degree of complementarity and substitutability existing between them.

To find an example of the questionnaire click here

Each respondent must evaluate a hypothetical scenario for each node (combination of indicators) assigning a score inside the [0,10] numerical scale that corresponds to the [extremely bad,very good] qualitative scale, respectively. When a particular node is composed of two indicators, two hypothetical scenarios must be evaluated; if there are three indicators, then five scenarios must be evaluated.

The new elicitation algorithm, developed for FEEM SI 2013 edition, has three advantages, since it:

a) allows fuzzy measures elicitation without making respondents aware of being in a fuzzy dimension;

b) guarantees the solution uniqueness;

c) c) returns a proxy of incoherence for each respondent that allows weighting according to his/her answer.

b) guarantees the solution uniqueness;

c) c) returns a proxy of incoherence for each respondent that allows weighting according to his/her answer.

**b) Experts' profile**

Since the concept of sustainability could depend, not only on the subjective preference of experts but also, on their geographic location, in each FEEM SI edition, we aim to involve the highest number of experts from different nationalities.

The FEEM SI 2013 edition involved experts from Belgium, France, Germany, Italy, Norway, Sweden, Turkey, USA, Ecuador and Australia with strong expertise in economic, social and/or environment science and working knowledge of different types of institutions, such as Academia, International Organisations and International Think Tanks.

**c) Aggregation of experts' evaluations**

Since the FEEM SI 2013 elicitation method can catch the incoherence of each respondent in judging a hypothetical set of scenarios, this important information has been used to weight his/her answers accordingly; this means that the more an individual turns out to be incoherent, the less his/her answer will be weighted, with respect to the others.

Hence, for each node of the decision tree, the aggregated/final set of measures will be the weighted average – according to the incoherence index - of each respondent's measures.

**d) Shapley values**

In the contest of fuzzy logic, the interpretation of fuzzy measures is not an easy task; for this reason some indices have been developed in order to give a direct, specific and overall interpretation of these measures. For example, the Shapley value represents the overall importance of a criteria.

Given the measures obtained in c), the following figure shows the Shapley values for each node of the FEEM SI 2013 decision tree:

Moreover, the relative importance of each indicator, with respect to the overall index, can be calculated by multiplying the Shapley values of all nodes located above the indicator under consideration.

Choquet methodology for aggregation

Choquet methodology for aggregation

Once all the measures have been derived and all the data inputs have been normalised (link alla sezione). Furthermore, all indicators belonging to the same node of the decision tree have been aggregated into a single numeric value by means of the Choquet Integral. This special type of aggregation tool is able to take into consideration, not only the relative importance of each indicator belonging to the same decision set but also, the degree of interaction, hence the degree of complementarity and substitutability existing between them.

**Sensitivity analysis**

The subjective preferences provided by experts, together with the structure of the decision tree, are crucial in the aggregation methodology of the FEEM SI; therefore, a sensitivity analysis has been carried out to assess, for each country and year, the volatility of the FEEM SI, due to a variation in the experts' preferences. "By means of Monte Carlo simulation, a huge number of hypothetical experts were generated"; the aggregation process has been run for each simulated preference in obtaining a distribution of the FEEM SI, and consequently, its fundamental statistical measures (mean, standard deviation, maximum and minimum).

##### Browse through FEEM SI publications to know more about the FEEM SI aggregation procedure and sensitivity analysis.

##### Detailed sensitivity analysis results are available in the Results section