Project Portfolio Management: Project Selection

IT outsourcing is a growing industry. It allows industries to focus on their core business while letting specialized companies manage their IT infrastructure, operations and application development. This strategy has been deployed with success in a number of organizations around the world allowing both sides of the partnership to grow. IT is thus deployed as a service.

When discussing about IT outsourcing, the main distinction is whether infrastructure and / or applications are outsourced. In the remaining part of the article, infrastructure outsourcing will be referred to as IO while application outsourcing will be referred to as AO.

• IO focuses on the IT components of the organization: from network backbone (routers, switches, firewall ...) to end-users workstations without forgetting servers (mail, web, ERP, …) or support. In addition of the daily operations of maintenance / upgrade / etc..., IO might as well include the implementation of new projects.
• AO on the other side takes care of the applications development and ensures that it supports the business in the daily work of the end-users (whether they are manufacturing employees, clerks, salesperson, etc...).

In the example that we consider, we analyze a case of a production company (Production Inc.) that outsources both infrastructure and application development to an external IT partner (IT Inc.). IT Inc. is therefore responsible of all operations and application development for Production Inc in a full IO and AO deal. In addition, IT Inc. is a key advisor of Production Inc. and, due to its deep knowledge of all IT operations, is in a good position to suggest potential improvements and new projects to implement the required changes. On a yearly basis, IT Inc. gathers inputs from all its employees in order to suggest to Production Inc. the best projects that will support its operations and growth. Furthermore, the executive team of Production Inc. also has a list of projects that they would like to conduct and therefore need to select which projects are more worth their investment.

In this context, Production Inc. needs to combine its own projects and the different projects submitted by IT Inc. and will need to evaluate which are the best. This evaluation cannot be only based on the sole costs (whether implementation, maintenance, training ...), thus making usually the selection of projects a complex task. In this particular scenario, Production Inc. has identified a number of criteria that they wish to consider before taking a decision on the projects to implement. Those criteria have been divided into two main categories: price and quality, each category being subdivided as shown below:

1. Costs
   1. Project: those criteria reflect the direct costs visible to Production Inc.
      1. Initial Investment: this is the cost of additional components that need to be added to the infrastructure (for instance new servers, new network routers ...). It is expressed in Euros.
      2. Implementation cost: this is the cost of labor necessary to implement the project (installation, configuration, integration ...). It is expressed in Euros.
   2. Operational: this reflects the costs once the project has been completed and that it is fully functional (i.e. put in production)
      1. Daily operations: this is the difference of costs between daily operations before the implementation of the project and once it is completed. If the system is less expensive to work with on a daily basis, it can therefore contain negative values. It is expressed in Euros.
      2. Knowledge transfer: this is the price of transferring the knowledge from the people who realized the project to the people who will maintain the solution in place. It is expressed in Euros.
2. Quality
   1. Logistic: this subcategory reflects the planning of the project in terms of required efforts and length
      1. Length: this is the expected length of the project, from the planning phase until it is deployed into the production environment. It is expressed in days.
      2. Man days: this is the required effort and it highlights how much time the different resources working on the project are going to spend. This is expressed in Man Days (MD) where 1 MD is the time spent by one resource during one day (8h). This criterion is not directly related to the length of the project as it depends of the time repartition.
   2. Strategy: this subcategory assesses how the projects align with the strategy defined by the executive team of Production Inc and what are the risks involved with the implementation of the project.
      1. Strategy alignment: this criterion measures against a qualitative scale how the project is aligned with the predefined strategy of Production Inc. The scale is [10%; 20%; 40%; 60%; 80%; 100%].
      2. Risks: the risks associated with this project is expressed against a qualitative scale [no risk; low; medium; high] defined internally by Production Inc.
   3. Service: this subcategory highlights how the projects are going to impact the level of service provided by IT Inc.
      1. Impact: the impact is measured against a qualitative scale [no impact; limited impact; high impact] and shows how the projects are going to impact the daily operations. This does not only include technical reasons (expected downtime, data unavailability…) but also the learning phase for the employees (new functionality, new product ...).
      2. Service Level Agreement (SLA): finally, Production Inc will assess whether a SLA will need to be put in place in order to ensure that the new service / system implemented complies with their expectations. The scale is [no SLA; above 90%; above 99%; above 99.9%].

D-Sight is launched by default with the five main inputs panels. The first one is the Evaluations panel gathering all the alternatives (Project A, Project B …) and all the criteria. They are respectively represented in each row and in each column. As represented in Figure 1 (click on the image to see the complete table), the evaluations can be numerical or based on qualitative scales.

The two following panels allow the decision maker to list the alternatives and criteria. Names and short names can also be assigned to all these elements. The fourth panel is meant to introduce the evaluation parameters and is represented in Figure 2 below that allows the interested reader to consult the parameters used in this case.

Figure 1 - Click to enlarge

Figure 2 - Click to enlarge

Figure 3 - Click to enlarge

The last of the five panels allows structuring the criteria into a hierarchy tree. Categories can be defined in order to gather criteria or other categories or both. This tree can easily be constructed by drag and dropping its elements. The categories are defined in that panel. As represented in Figure 3, each category has also a name, a short name and a color. They have two additional interesting properties. The first one is that they can be disabled as the criteria or alternatives. The exception in that case is that when a category is enabled / disabled, all its content is automatically enabled / disabled. The last property of a category is that it can be grouped or ungrouped. This will be illustrated later on.

The Global Visual Analysis offers a complete representation of the problem. As shown in the below figures, the alternatives are represented by the blue points and the criteria are represented by the different axes. Let us note that the criteria are not represented “as is” but according to the decision maker’s preferences (e.g. the preferences functions). The computation of this chart depends on two things:

• the alternatives and their gross criteria evaluations;
• the intracriterion parameters expressing the way the decision maker wants to evaluate those alternatives for the different criteria.

As the decision maker has here defined a hierarchy, he can choose what he wants to see in the GAIA plane by grouping or ungrouping the categories. In the three following figures, the Costs and Quality categories are ungrouped. Their respective category children are represented as they are grouped. Thanks to this plane and the “grouping” functionality, the alternatives can be easily compared on different level of details. Let us remind the reader that different types of observations can now be done:

Figure 4 - Click to enlarge

• Relative positions of the alternatives: if two projects are close in the plane, as J and B, it means that they should be similar with respect to the different criteria.
• Alternatives and criteria: an axis indicates the direction of the area in which the alternatives are good for the associated criterion. In the seconde figure below, the decision maker can directly see that F is the best alternative on the global Logistics (e.g. F goes the further on the Logistics axis).
• Relative positions of the criteria: if two criteria axes are in the same direction, that means that they are similar (e.g. globally, the projects which are good for the first criterion are also good for the second). If two criteria axes are in opposite directions, that means that they are conflicting (e.g. globally, the projects which are good for the first criterion are not good for the second). In the following illustration, one can observe that in average, the projects having interesting Strategy scores are not good on the Operational aspects (e.g. Strategy goes on the left while Operational is oriented to the right).

The red axis (the one without label) of the GAIA plane is called the decision axis. It is obtained by using the weights of the criteria given by the decision maker. It represents the compromise he wants to make between the criteria. By projecting the projects on this axis, the global ranking can be visualized as shown hereunder.

We would like to emphasize that the ranking inferred visually by projecting the alternatives on the criteria axes or on the decision axis may not be completely correct. This is due to the fact that a two-dimensional representation of a k-dimensional problem is used. For more information, please consult the complete PDF article.

Figure 5 - Click to enlarge

Figure 6 - Click to enlarge

The PROMETHEE II ranking offers a total ranking of the alternatives from the best (the first rank) to the worst (the last rank). They are scored between 1 and -1; 1 being the best. The ranking of the projects is represented in the illustration hereunder. Project F is first with a score of 0.20, closely followed by project K, with a score of 0.19.

Figure 7 - Click to enlarge

In the previous section, we mentioned that the visual global ranking can be obtained by projecting the alternatives on the decision axis. In the figure hereunder, we have grouped the two main categories, Quality and Costs, which allows avoiding any loss of information (as working in 2 dimensions). The delta value is, in that case equal to 100%. One can now observe that the projections of the alternatives on the decision axis do perfectly reflect the PROMETHEE II ranking, with the ordinal and cardinal information.

Figure 8 - Click to enlarge

Another interesting point related to this visual configuration, is that it allows quantifying easily the Quality / Cost trade-off that could be done by choosing a project over another. Indeed, one can observe that choosing between F and K, which are almost equally scored in PROMETHEE II, is equivalent to choose between Quality and Costs.

In most multi-criteria analysis, the assessment of criteria weights is a crucial step. Indeed, these may deeply influence the global score calculation. Therefore, it is really important to be able to perform sensitivity analysis. In D-Sight, this is done by using the stability intervals. They give, for each criterion or category (according to the decision maker’s choice), the interval in which the weights may-vary without affecting the ranking.

Figure 9 - Click to enlarge

We can observe in figure opposite, that some of these values are rather robust while others are more sensible. Let us emphasize an important point here: one could think that the Logistics category is stable because the weight can increase from 18% to 100% without affecting the ranking (to the level 1). But on the other hand, it cannot even be decreased by 1% without affecting the ranking. That means that Logistics is only stable “on the right side”. This table shows us that there exists some asymmetry in the stability. Let’s highlight that the following hypothesis is here considered: when a weight is changed, the other weights are proportionally changed, with respect to the initial ratios. The stability level is the rank to which the stability is considered. In the present case, the level is one which means that we only look at the first alternative and in which intervals it stays first, regardless if the third alternative becomes second. The stability level can be calculated up to n. The bigger the level is, the smaller are the intervals, as constraints are added.

The previous analysis has led us to identify a sub group of alternatives that are more interesting than others. Clearly, F and K seem to be good candidates. A natural next step is thus to focus ourselves on the differentiation between these two options. The profiles of the alternatives can be compared two by two by using the Profiles tool. In the following figure, F and K are being compared for all the level-2 categories. It is easy to see that they do not have similar profiles, especially for the Project, Logistics and the Strategy criteria.

The same type of analysis can be made in a spider-web chart, as represented hereunder. In that case, we compare the two first alternatives, F and K, for all the sub criteria (level 3). We have just ungrouped all the categories, which has the consequence to automatically update the chart.

Thanks to this chart, one can observe that K (K is the one that is best on the Risks) has a more averagely good profile. On the other hand, F has strong points while it is very weak on others.

Figure 10 - Click to enlarge

Figure 11 - Click to enlarge

We will now consider the case in which the decision maker has to select several projects. He thus needs to establish the best project portfolio. Having ranked all the projects by attractiveness, the decision maker will have to determine which projects he will select in order to respect some imposed constraints.

Literally, this problem consists in selecting the best subset of alternatives (with respect to their global scores) while considering different constraints. To illustrate this, we will use the sub-set optimization plugin that is available for D-Sight. This plugin is based on the PROMETHEE V method.

The process is divided in two steps:

1. A ranking of the alternatives / projects is established, considering the preferences of the decision maker. Thus they are ranked with respect to their attractiveness for the decision maker without taking into account, any constraints or relation between the alternatives.
2. The decision maker can provide different constraints such as a budget or a minimum number of alternatives having to be selected, for instance. A subset of alternatives is then found to maximize the total additive score of the subset while respecting all the established constraints.

We will now come back to the example in order to illustrate the use of this plugin. Let’s consider that Production Inc. wants to implement different projects but needs to respect the following constraints:

• A limited Initial investment of 750.000 €.
• A limited Implementation Cost of 1.500.000€.
• The number of selected projects must be greater or equal to 5.
• If project A is selected, project I must be selected.
• If project I is selected, project C can’t be and vice-versa.

The different constraints can be defined into an interface as illustrated in the following figure.

Figure 12 - Click to enlarge

Figure 13 - Click to enlarge

As the constraints are being added, the subset is gradually computed taking into account the additional constraints. Finally, if there is at least an existing solution, the optimal subset is calculated. As illustrated hereunder, the selected alternatives are highlighted in green in the table. The “Show Everywhere” button allows displaying the green (selected) and red (not selected) colors in all the charts and tables of the software. This can help to integrate the optimization results in the other tools in order to make an easy global analysis.

Let’s emphasize the fact that many types of constraints can be defined. Furthermore, custom groups can also be defined in order to use group related constraints.

In this section, we will consider the case in which two decision makers are involved in the projects selection. Let us consider two managers, John and Jane.

As defined in the model, each participant works with the same set of alternatives and criteria (structured in the same way).

The participants can differentiate their analysis on three main points:

• The evaluation of the alternatives for the criteria: this case is relevant for criteria based on qualitative judgments given by different experts.
• The evaluation parameters: the actors can decide not to evaluate the criteria in the same manner. They can have different functions or different thresholds.
• The weights of the criteria: each stakeholder can weight the criteria according to his own preferences.

In the previous part of this article, John was the only decision maker in the selection process. We will consider that John and Jane will only weight the criteria differently. Indeed, Jane accords more importance to the Costs than to the Quality. She also has given different weights to the sub-categories. In the left figure below, we have represented the visual analysis and the global ranking according to John’s preferences. We logically have the results previously presented.

Figure 14 - Click to enlarge

Figure 15 - Click to enlarge

To change the current decision-maker, one has just to select the desired user in the list. The entire interface is automatically updated with the corresponding data and results. In the right illustration above, Jane’s results are represented. The ranking of the alternatives is different and so is the GAIA plane. The reader could expect the GAIA plane to be the same in the two cases, except for the decision axis, as only the weights have changed. As the weights of the sub-categories are not the same for John and Jane, the unicriterion scores of the alternatives for the Quality and the Costs are different which explains the computation of two different planes.

As we have now had a look at each stakeholder’s results, we can now analyze the global results. A new dedicated window is open and is organized the same way than the main one. It is represented in the following figure. Its look and feel can be set differently to be easily recognized by the user.

In the evaluation table, the projects are still represented in the rows but the columns do not represent the criteria anymore. Indeed, each column represents a decision maker. That means that this table contains the scores of the projects for the different stakeholders.

As each stakeholder is weighted, that allows computing a global aggregated score for all the projects. We can see in the following figure that the project F is the most preferred project while taking into account each stakeholder’s point of view. Furthermore, a GAIA plane can also be represented with the aggregated users. In this representation, the points are used for the projects but, as expected, the axes now represent the actors. The decision axis (the axis without label), is computed with respect to the weights given to the decision makers in order to represent the direction of the most globally preferred project.

If we take a closer look at the GAIA plane, we can directly see that there is a group of alternatives, situated in the left part that will never be chosen if both decision makers have to agree on the solution.

In the right part of the plane, two different subsets of projects are observable (represented below in more details with the projections):

• The conflicting projects (K, C, H): those solutions are well scored but are not preferred the same way by each stakeholders. For instance, project K is very good according to John but is only fourth according to Jane. In general, conflicting alternatives have crossing lines when projecting on the decision makers’ axes.
• The best consensus projects (project F): those solutions are preferred by all the decision makers. They have no crossing lines when projecting on the decision makers’ axis.

Figure 16 - Click to enlarge

Figure 17 - Click to enlarge

The multi-actors plugin will be soon extended to an online group decision support system. This will allow the stakeholders to work remotely as well as having different roles (e.g. project manager, expert …) in the decision process.