Solution and Methodology | Political Harmony Consulting Services

Solution Background

Dr. Ray Garubo, through years of local government consulting, recognized the need for a statistically reliable tool to quantify intra-organizational effectiveness, leading to the creation of the Civic Leadership Assessment. The Civic Leadership Assessment is an online 31-item assessment tool that enables public management, staff, and the public to be evaluated using a statistically reliable analytical model.

The model focuses on six primary ‘dimensions’ that will reflect the efficiency, coordination, management, and effectiveness of the assessment subject and its image. Each of the three constituencies’ scores will be compared and contrasted against the other two, and also to national averages. This provides a multi-layered view of areas of agreement or potential conflict across each ‘dimension’.

The resulting product, the Assessment Report, based on the findings in the completed assessment forms, identifies areas of agreement (harmony) and more importantly, areas of divergent opinion and areas for improvement. Where appropriate, recommendations will be made as to the next steps to a more effective and conducive working environment and relationships.

The Six Dimensions of Political Harmony

The Civic Leadership Assessment scores the six main components relevant to any City Councils’ successful and effective functioning. Those components are:

Effective Communication
Effective Conflict Management
Ethics and Public Trust
Requisite Knowledge
Financial Management
Effective Leadership

The Civic Leadership Assessment contains includes questions related to each of the six components, for a total of 31 questions. The score for any given component is the arithmetical average of the items that compose it.

An example of a assessment question is: “The budget is well managed” which is part of the “Financial Management” component.

The methodology employed to create the Civic Leadership Assessment began with simple, subjective, face-validated efforts and culminated in empirical, quantified, and verifiable analyses.

What is the Civic Leadership Assessment?

The Civic Leadership Assessment was conceived, developed, refined, and tested for reliability and validity. What follows is an account of the evolution of the Civic Leadership Assessment, and an outline of the subjective and objective steps undertaken to create the assessment.

Dr. Ray Garubo is the creative force in the development of this assessment. He has been employed as a public school teacher, public school administrator, professor of education and public administration, author, consultant, and doctoral advisor. The components of “harmony” come from years of teaching, researching the literature, and finding the best practices from case analyses.

A national random sample of school districts and city councils conducted from 2003 to 2006 revealed that the assessment was reliable and most likely measured at least the core ingredients of harmonious interaction among elected officials with differing circumstances, agendas, and political persuasions. Ongoing research is still being conducted and a national database is being constructed as assessment forms are analyzed.

Civic Leadership Assessment Format

The Civic Leadership Assessment is an online, self-administered instrument (see it here). A respondent identifies themselves as, 1) an elected official, 2) a staff member or 3) a member of the general public. There are 31 statements to which a respondent chooses one of a six-point Likert-type scale which ranges from “Strongly Disagree” to “Strongly Agree.” There is also a “No Opinion” option which receives no weight. The total score for any given component is the arithmetical average of its five items where “Strongly Disagree” is assigned a score of “1” and “Strongly Agree” is scored as a “6” with equal unitary intervals in between. The items are presented in such an order that it would be highly unlikely that a respondent could divine that there are 6 separate components and what those components are. Some statements are fairly transparent but others are not readily discernible.

Data Analysis Process

Currently, the raw data for each respondent is collected online into a SQL database and is exported from the database and input into an Excel spreadsheet to create the anonymized Assessment Report. During the development of the original model, hard-copy surveys were data entered into Excel. Most of the descriptive statistics, e.g. averages, standard deviations, simple correlations, charts, etc. were performed by Excel. The more complex relational statistics that validate the statistical integrity, e.g. split-half reliability, factor analysis, Cronbach’s alpha, multiple regression equations, etc. were modeled using the Statistical Package for the Social Sciences (SPSS) version 10.

The Pilot Study

To validate and assess the reliability of the Civic Leadership Assessment, a pilot study was conducted in which a minimum of 25 people from each of these three categories were included: elected official, staff, and public.

The assessment employed a Likert-type scale that contains “equally appearing intervals” and no true zero.

Furthermore, histograms for the 30 individual items were computed to assess how close they were to approximating a “normal” (bell-shaped) curve. The items were reasonably close to “normal” and thus the two fundamental requirements to perform parametric statistics were met and the analyses proceeded legitimately.

The pilot study statistical analyses began by computing mathematical averages and standard deviations for all 30 items and their 6 respective components for all respondents. Next, means and standard deviations for the six components were computed for the three responding categories; elected official, staff, and public.

The table below displays the results for the elected official sample of 33, with a maximum score of 6.0 for any mean. The overall average for all six components was almost exactly 5.0 or “Agree.” The standard deviations were quite typical for scales of one unit.

Elected Officials	Mean	Standard Deviation
Effective Communication	5.2	0.8
Effective Conflict Management	4.9	0.7
Ethics and Public Trust	4.9	0.7
Requisite Knowledge	5.1	0.6
Financial Management	4.9	0.8
Appropriate Leadership	5.1	0.8

The means and standard deviations for the Staff and Public samples were also highly similar and slightly lower but not significantly so.

A 3 X 6 Analysis of Variance compared the three responding categories on the 6 components of civic harmony and addressed the statistical question: Are there any significant differences between the categories, between the components or among the eighteen category/component combinations? If the ANOVA is not significant that result means that there are no differences anywhere. If the ANOVA is significant, that result confirms that there are significant differences somewhere between the collective groups but it does not identify which specific components created those differences. The computed ANOVA was not significant at the p <0.05 level. Thus there were no significant differences between the three respondent groups or the six components.

Reliability and Validation

The Civic Leadership Assessment has 30 items across 6 components (dimensions) and a total of 31 statements (assessment items). To objectively evaluate the degree to which each item accurately measures its respective component, Cronbach’s alpha was computed for each item. “Cronbach’s alpha measures how well a set of items (or variables) measures a single unidimensional latent construct. If the inter-item correlations are high, then there is evidence that the items are measuring the same underlying construct. Cronbach’s alpha is a coefficient of reliability or consistency.” (www.SPSSFAQ.com) Cronbach’s alpha ranges from zero (no inter-relatedness at all) to 1.0 (a perfect relatedness).

To determine which items should be retained and which ones should be deleted, if any, Cronbach’s alphas were computed for each individual item for each component to assess the degree of inter-relatedness if that item was removed from the set of five. Usually to improve something we think in terms of adding something. The Cronbach’s alpha process is kind of backward in that something is improved by removing the weak link. By this well-accepted procedure by statisticians, the items relate to each other in meaningful ways. The Report will compare and contrast individual items from certain dimensions to others, for example, Communication to Ethics and Public Trust.

All 30 items yielded a Cronbach’s alpha of at least 0.8. The average alpha was nearly 0.85 which is extremely large. Therefore all 30 items were retained and provide testimony for a highly reliable instrument.

Factor Structure

The Civic Leadership Assessment contains 6 components that were designed and believed to be separate and distinct elements of local government and school administration. Factor Analysis is a method that endeavors to extract fundamental themes, commonalities or subsets from a larger set of variables. The found themes are called factors and an individual variable’s correlation to the factor determines to what extent it lies within that factor. A variable’s correlation to a factor is called a “factor loading.” Factor loadings of .7 or greater are considered meaningfully large because nearly 50% of that variable is contained in that factor.

To explore the factor structure of the Civic Leadership Assessment, a principal components factor analysis, rotated to the varimax solution was performed by the SPSS program on the 6 components. Although the six components of harmony appear to be relatively independent of each other, it is the nature of psycho-social research that many aspects of human functioning are highly interrelated. Therefore, it would not be at all surprising if a factor analysis of the assessment yields something less than 6 different and unique themes.

The results of the Factor Analysis yielded just one, very large factor which contained all six components with big factor loadings of 0.84 or larger. The logical conclusion then is that the six components are tapping into six different aspects of the same construct. The overall abstraction for the assessment is “Civic or Political Harmony.” Since it seems that the six components measure a different aspect of one larger entity, whatever label that encompasses all six components is what this assessment is about. All six components are part and parcel of one larger concept. “Political Harmony” is one potential descriptor of that concept.

Regression Analyses

A regression equation is a statistical method that mathematically combines any number of independent variables (predictors) to yield the maximal correlation to a single dependent variable (the criterion.). One type of regression equation is built by first selecting the independent variable which has the largest correlation to the dependent variable. In the second “step,” another independent variable is mathematically selected such that when that second independent variable is combined with the first one, the inter-correlation between the combined independent variables and the dependent variable increases significantly. Other steps are possible as long as additional independent variables added to the equation continue to increase the overall correlation significantly.

The last item in the Civic Leadership Assessment is: “There is adequate political harmony among the Board/Council.” The response to this item served as a measure of the Board/Council’s overall performance. It was a dependent variable (or criterion) predicted in two ways. First, a stepwise, multiple, linear regression equation was built employing the six components of harmony as the predictors of the overall performance. This equation revealed which components when weighted optimally yielded the greatest possible correlation to the overall variable and the size of that correlation. Secondly, a stepwise, multiple, linear regression equation was built employing the 30 individual items as potential predictors of the criterion. This equation also revealed which items when optimally combined yield the highest correlation to overall performance. Regression equations are often helpful in identifying what behaviors are contributing to an organization’s overall success. They also aid in pinpointing the differences between officials vs. staff vs. public views of what is being done well and what is not.

The best single predictor of Political Harmony was Ethics and Public Trust with an extremely large correlation of 0.82. Thus, Ethics and Public Trust account for nearly 66% of the variation in Political Harmony. Three other components, namely, Requisite Knowledge, Conflict Management and Communication also contributed to the prediction of Political Harmony but each individually only to a minor degree. These 4 components collectively, however, accounted for 83% of Political Harmony. It is common for psychosocial research to have as much as 25% to 30% of the criterion variance to go unexplained by the predictors. So, a relatively large amount of Political Harmony is understood and accounted for.

A second regression equation employing all 30 items to predict Political Harmony mirrored the components’ regression results, almost exactly. There were two items from the Ethics and Public Trust component and one each from the Requisite Knowledge, Conflict Management, and Communication components. The order of the items was slightly different but the content was identical. Again the percentage of variance accounted for by the four items was quite large for individual items.

In combination, these two regression equations again emphatically demonstrate the reliability of the assessment and its ability to measure what it purports to measure.

Reliability and Pilot Assessment Summary

The reliability of the Civic Leadership Assessment was empirically quantified by various mathematical procedures and analyses.

There were no significant differences between the Elected Officials, Staff, and the Public. The Public scores were lower but not significantly.

All 30 of the original items were tested for reliability via Cronbach’s alpha and found to be highly significant. Therefore none of the 30 items were removed or replaced from the assessment.

A factor analysis of the six components yielded one large factor that included all the components to a meaningful degree. Thus the six components are part and parcel of some larger construct. Political Harmony is at least a potential label for that construct.

By far the largest component contributor to Political Harmony was Ethics and Public Trust as established via a multiple regression equation. Requisite Knowledge, Conflict Management, and Communication also contributed to a far lesser degree.

Individual items from these same components were also significant predictors of Political Harmony.

Reliability can be demonstrated and calculated with considerable mathematical precision. The validity, however, is not so easily quantified. There are no known comparable instruments with proven validity. So there is nothing to serve as a comparable guideline. The Cronbach’s alphas showed a high degree of consistency, so for any given component the items are at least addressing the same thing. The fact that the content of the items certainly appears to be addressing the same thing as well, then the component is most probably face valid.

Additional Normative Assessment and Findings

Subsequently, a much larger sample was obtained. There were 311 participants altogether. In this assessment, approximately 66% were mayors or city council members. About a third were school board officials. Thirty-six were “staff.” There were no “public” participants.

All of the statistical findings reported above were replicated by this larger sample.
The overall averages for all six dimensions were within one-tenth of the pilot sample. Likewise, the mayors and school officials did not differ significantly on any dimension.
The standard deviations were all smaller as would be expected as the sample size increases.
The country was broken down into six regions: Northeast, East, South, Mid-West, West, and Pacific. The East had slightly higher scores, but all other regions did not differ from one another.
The “staff” gave lower ratings on Requisite Knowledge and Adequate Leadership than the officials gave themselves. All other dimensions were very similar.
The item analyses via the Cronbach alpha statistic reaffirmed the reliability of the employed items and their contribution to the six dimensions.
The factor analysis again found one large factor with all the dimensions and the Overall Harmony variable loading very highly on that single factor.
The regression equation also again showed Ethics and Public Trust to be the single best predictor of Overall Harmony. Two other dimensions contributed to the prediction of Overall Harmony but to a minuscule extent.
The vast majority of these participants were from relatively small towns and cities.
There were insufficient data to perform any comparison by city size.
A sample size of over 300 subjects provides a very stable and reliable database. It is highly unlikely that even a much larger sample size would differ significantly from what has already been found.

Conclusion

The Civic Leadership Assessment is proven statistically reliable for an assessment size of 10 or more respondents per category: elected officials, the staff, and the public. Any organization or business entity that involves leadership, finances, and teamwork can benefit from the assessment.

The Civic Leadership Assessment is a valuable tool for any group interested in understanding the potential differences in perception across organizations or constituencies Particularly those that are interested in finding a way to improve the workings and effectiveness across them. And the Civic Leadership Assessment provides an extremely cost-effective, and minimally intrusive way to achieve this.

If you have any questions about how the Civic Leadership Assessment can help drive improved communications and organizational effectiveness, please contact us.