Chris Park outlines what went into the making of a valid performance measurement system for the Dublin BIC
I was a member of the Board of Dublin Business Innovation Centre (BIC) for many years and for most of that time I was frustrated by the inability to objectively measure performance of the organisation beyond the internal data provided. This was not, by any means, the fault of the management but by the lack of external data by which to check progress or otherwise.
Normally commercial profit organisations have measures such as profit before tax, or return on investment etc, which can be tracked internally over time and, by reference, to similar published information on the relevant industry. But these measures are meaningless in the case of a BIC. Consequently it was necessary to construct other measures to evaluate performance. In addition, government agencies and other organisations which fund the BIC have economic objectives such as business start-ups, job creation, improving technology etc, rather than a profit motive. While these measures were used internally within the BIC, it still required some external measures to determine if we were doing the right thing. If, for example, we created say twelve start-ups last year and 10 the year before, was the 20 percent increase something to be proud of when other similar BICs were doing an average of, say, 20 start-ups a year? We just didn't know what the standard should be and what we should be striving for.
The arrival of the European BIC Network’s (EBN) EC-BIC Observatory some years ago provided information upon which to build a valid comparison model and evaluate individual BIC performance. At last it was possible to compare like-with-like and there is now an extensive database from which specific data can be extracted for further analysis. The observatory published averages and sometimes the medians for a range of data, including enquiries, feasibility studies, business plans, start-ups and jobs created etc, for 90 percent of the BICs in the network in 2011. Sadly the proportion of BICs participating in the 2012 survey has dropped to 84 percent of the total network. Hopefully this number can be increased in 2013.
Data available and the limitations
A benchmarking exercise of Dublin BIC (using averages and medians provided in the BIC Observatory), while very useful has certain limitations and this gave rise to doubts among some about the efficacy of the exercise. The inclusion of BICs in some countries in the data might skew the data and make a good comparison more difficult.
For instance, including BICs from Eastern Europe, such as Poland or Slovakia, which are developing economies with different cost structures might have a distorting effect on the averages. In order to overcome this potential conflict and assure everybody that any analysis and conclusion from the benchmarking exercise was valid it was decided to ask the European BIC Network to provide selected data from a random selection of ten or twelve BICs which fell within certain parameters.
The parameters decided on were:
- Total expenditure: €2 - 3 million
- Staff Numbers: Twelve to thirty FTE Staff (full-time equivalent)
- Clients: Mainly hi-tech or knowledge based companies with 70 percent or so of
- client companies focused on technological innovation
- Western Europe: Exclude Eastern Europe
- Catchment area: One to three million people
The parameters had to be reasonably wide, and not too restrictive, in order to give EBN scope to select at least ten to twelve BICs which came within or close to the parameters. In supplying the data required, EBN did not identify the BICs involved, it only gave each individual BIC a number and the country of location. This means that the data was not identified and would remain so unless the relevant BIC agreed to disclose its identity. No effort was made by Dublin BIC to trace any particular BIC except through EBN.
Type of data, organisation, analysis and Presentation
The question was; what data to request from EBN for the benchmark BICs? About 25 pieces of data for each benchmark BIC was requested, including such information as population of catchment area, promotional events organised, number of enquiries, new business plans produced in a year, total start-ups, jobs created and tenants in incubator, etc. In fact most of the data supplied in the BIC Observatory was included. In addition, aggregate data on the four Irish BICs was also requested. We were conscious of the need to limit the data requested so as not to overburden EBN, and also not to present too much information to recipients, including the Board of Dublin BIC. Too much data is worse than too little, especially for busy members of the Board who have to study a substantial amount of documentation before a Board meeting. ‘Data overload’, as it is sometimes referred to, can be self-defeating. Consequently, data was limited mainly to measures of performance and productivity.
The data was then organised and analysed using a Microsoft Excel spreadsheet to produce averages and medians of the twelve benchmark BICs, as well as averages for the network as a whole and averages for the four Irish BICs. This information was then graphed and transferred to PowerPoint for presentation purposes. It was decided that the best type of graph for this purpose was a bar chart, where Dublin BIC's data could be easily compared with the individual twelve benchmark BICs and the averages and medians of that group as well as the averages for the network and Irish BICs. The presentation was divided into five sections: Promotion and Training, Operations, Value for Money, Productivity, and Dublin BICs performance over the previous five years.
Results and conclusions
While Dublin BIC compared favourably with the averages for the benchmark group on a range of measures, there were two BICs in the group which performed exceptionally well, one in Italy and one in France. The question was: how did these two BICs manage to do so well? To confirm the position and to make sure that the result was not a ‘one off’ performance we asked EBN to supply data for the previous two-three years for the two BICs in question, without of course, disclosing their identities. The results showed consistency in performance, indeed an improvement in these measures. Consequently the Board and management of Dublin BIC were very interested in the reasons for the exceptional performance which it wished to emulate if possible. EBN was asked to contact the BICs in question and to seek to arrange a visit to each BIC to discuss operations and share experiences. One BIC responded soon after and agreed a visit, while the other asked to postpone a visit to a later time. This was the first time a benchmark BIC was identified to Dublin BIC and a visit to a BIC in Italy took place in July 2013.
Before the visit, Dublin BIC disclosed all its data to the Italian BIC so that both shared similar information. In the event, it was discovered that the very high number of plans and start-ups compared to Dublin BIC and the overall high performance of the Italian BIC was mostly due to the large number of self-employed clients it helped to get started under a scheme promoted by the Italian government. It was apparent that the difference in performance was mainly due to this factor. Dublin BIC concentrates its efforts mainly on high-tech start-ups of companies, each of which employs several technologically-skilled people. All was not what it seemed from a first analysis of the data available, and it is sometimes necessary to look behind the figures.
However many benefits have emerged from the visit, principally from an agreement to link clients of Dublin BIC with clients of the Italian BIC. A number of high-tech companies of the Italian BIC were visited and brochures and other details collected with a view to interest some of the Dublin BIC clients in making a linkage.
Spin-off and improvements
A spin-off benefit of the benchmarking exercise is the fact that we, in Dublin BIC, have now collected data on our performance over a number of years and have analysed it and graphed the results. This information is used to track our performance over the years and helps in setting targets for the future. Other spin-offs are changes and improvements to the questionnaire, in co-operation with EBN, and getting more clarity with definitions, etc. For instance, the original question about the estimated time (in months) from first contact with the entrepreneur to the establishment of the company has now been re-phrased to read “… to the start-up of the business:” While the original phrasing of the question would appear to evoke the intended response it was taken literally by some and the ‘wrong’ answer was given.
There is also a difficulty with some as to when a ‘start-up’ should be recorded. What level of sales should be reached to qualify as a ‘start-up’? The criteria will differ from BIC to BIC, but the view taken by the benchmarking exercise was, as long as the survival rate is high (75 percent plus) then it doesn't matter if the start-up is recorded in the previous or the current year. And the overall survival rate for BIC companies is very high.
A similar issue arises in defining an ‘enquiry’ - contacts for enterprise creation. Some BICs record numbers in excess of 400, while others record numbers in just double figures (less than 50); the average for the network in 2012 was 263. Is a simple telephone call which results in a redirection somewhere else recorded as an enquiry, or must the potential client receive more in-depth attention before moving to the next stage?
The data used in the benchmarking exercise comes exclusively from the database of EBN, so it is important that the data supplied by each BIC to EBN is complete and accurate otherwise the information cannot be relied on. Completing the annual questionnaire is a major task. There are nine sections and over 170 questions to be answered, with many questions requiring multiple answers. While many questions require just a tick many others need a certain amount of calculation to be done before responding. For instance, calculating the number of full-time equivalent staff (FTE Staff) can be done accurately (based on the number of hours worked) or may be just a guess. Again the number of months from first contact to start-up may be a guesstimate whereas it can be accurately calculated with a little effort.
To aid the completion of the questionnaire we have devised a few simple Excel programmes to enable the operative to produce accurate figures quickly for FTE staff, estimated time (in months) from contact to start-up, income and expenditure, key sectors etc.
Cooperation and Support
The benchmarking exercise could only be done with the full co-operation and active support of EBN and we are grateful to Giordano Dichter and his team for all their help in providing the data in a timely manner. The Board and management of Dublin BIC have fully adopted benchmarking as a valuable tool in managing the BIC. It is now possible to form better judgements of performance and to set targets which, while challenging, are realistic by comparison to similar BICs elsewhere. The model can be applied to other organisations seeking to establish similar methods of systemisation - we can certainly vouch for its efficacy.
Published on 21-10-2013 12:35 by David Tee. 1243 page views
Back to TBI list