Wednesday, January 30, 2008

Farewell to Mangochi



We returned to the office and synchronized our data for review. The only problem noted was one set of PDA records seemed to have happened the night before. I chuckled when I realized that it was my mistake. I had forgotten to change the PDA's clock to the correct time zone after leaving Armenia last week. The PDA catches everything - including the consultants' mistakes!

That afternoon, I gave my final presentation to senior staff. I recommended that they procure several PDA's and use them in a pilot program in a parallel effort with a paper-based survey. I also risked an internal organizational suggestion. When I was picked up at the Nkpola Lodge on my first morning of instruction, my driver was actually Lloyd, the manager of Logistics and all of the drivers. Later that day when I asked to see where the servers are located, I was somewhat alarmed to see Lloyd sitting at the keyboard of one of the servers. I found that Lloyd was also the IT support for the Mangochi office. During our many drives together, I found Lloyd not only quite friendly but well skilled for IT work. I spoke with him and asked what would he really like to do - manage drivers or do IT work (he actually used to be an IT instructor in networks at a local school)? He replied IT. Kwandani, the Field Office Regional Technician (FORT) told me that Lloyd was doing a great job. So, during my exit presentation, I added a single bullet that the Mangochi IT support be a dedicated position. All were happy.

Field Testing at the Kausi and Nasite Schools - Mangochi



After the form was deemed designed, I taught the team how to load (also known as "synchronize") the form onto each of the PDA's that we would use for testing. Then, I had each PDA holder to conduct several interviews with other team members. We then synchronized the collected data back to the form designer and it looked all good. We were ready.

We took two vehicles and split the team in two and each group headed to a different school. The group that I went with arrived at the Nasite elementary school but a few kilometers south of Mangochi. I love this part. We first presented ourselves to the head teacher and it was explained why we were there, and what the purpose of our trip was.



While the team split up to interview teachers in their classes, I took a moment to enter a large classroom next door. A young teacher was seated at a table in the far corner of this room with a dirt floor interspersed with broken concrete and no lighting. He had several students at his desk as he was writing in a register. I asked him how many students that he had in his classroom each day, and he said that it averaged about 198.



As I walked out side his classroom preparing to follow and observe a PDA team member, I noted a group of children under a large tree trying to avoid the rain. The next series of pictures were taken within an interval of perhaps 5 seconds between them and within a total of less than 30 seconds:



2nd



3rd



The interviews went smoothly and we soon departed to meet up with the 2nd team at the Kausi school nearby. I will never forget riding up to the school site in the back seat and Amy Richmond in the front. Both of us are quite light-skinned. The chanting that we heard as we drove up across the compound was from the children racing across the yard saying, "The Whites are here, the whites are here!" As we were introduced to the head teacher (in a stunning bright blue suit), I noted several older students using small wispy branches to literally whisk the students back to their classrooms. I thought to myself, I wish I could of had that job back at my small elementary school in the 1960's suburbs of Southern California.

I was observing one of the team members interviewing a teacher with his PDA just outside the classroom. I emphasize the need to observe a respondents visual cues in order ot assess their willingness to be interviewed with technology, and to be alert on when there is a need to employ ones' developed set of interviewer skills to enjoin trust in order to have a successful, unbiased interview (if that is possible).



In the first picture above, the respondent is displaying body language of hesitation towards the interviewer and the PDA. Note the hands on the hips, and the leaning away from the interviewer.



In the second picture the interviewer has wisely turned the PDA screen towards the teacher, who has now begun to lean towards the interviewer and has begun demonstrating curiosity towards the interview collection instrument. Hold on, we're not there yet!



Now, the interviewer has gained the respondents trust and curiosity has been disregarded for the moment. note how the body language of the teacher has changed and now the PDA is disregarded and the teacher is instead thoughtfully considering his answers to the interviewers questions.

Please note, that with a PDA, an interviewer is not able to hold up the classic large clipboard and paper survey form which can often act as a barrier, both physically and communicatively between the two persons. A PDA is different and actually can be leveraged to enjoin trust.

Tuesday, January 29, 2008

What Will be the Results of My Questions?

The Results Framework

A note about the Results Framework. It is just that, a "programme logic that explains how the development objective is to be achieved, including causal relationships and underlying assumptions...” Wow! Let me give a simple and possibly error-prone example. Let's say you have suddenly had some unexpected even in your life that caused to perform some introspective review of your life's' goals. Let's say further that you lost your job and did not know how you would pay the bills and buy food. You might first think that you need to quickly get another job. But, you realize this might be an opportunity to review your life in general. You do just that, move it up a few levels and generalize. What do you want in life? Your responses might be, "I want to be happy!" Well, take it down a level. "I want to have money to buy necessities and enjoy good health."

Go further. "What will I do to enjoy good health (eat right, exercise, maintain healthy social relationships, etc.). Well, an important part might be, "how will I know that I have achieved these goals?" Well, you might define a list of measurements such as ideal weight attainment, percentage of healthy food groups in each weekly meal, defined progressive weight changes, blood pressure, cholesterol, etc. measurements over the year. As you disciplined yourself to measure these criteria of having good health, which in turn would be an element towards achieving "happiness", you would have an idea if your program of self-improvement was working...or not working. If your program was not working to the desired goals, you would either review or modify those goals (I really cannot drink another drop of organic carrot juice!), or change them altogether ("weight loss is not really producing the desired effects!).

My illustration is the questions that you would continually ask yourself would be based on your original objectives and the criteria that you defined to ensure that you were headed in the right direction to achieve that "state of happiness". The vital issue is not how healthy am I, but am I "happy".

The Results Framework is a method of defining a program's objectives, defining a strategy to achieve that objective and then to describe intermediate results that will yield indicators to tell you if you are indeed achieving the "state of happiness" for the programs beneficiaries. Likewise, in relation to survey questionnaires, it is not the question itself that is important, but what indicator was defined by the Results Framework that is described by the question so that I might monitor and evaluate whether I am achieving my strategic objectives for the programs success for the community.

So, how would the Results Framework help me build better survey forms? A clear definition of what I am trying to achieve towards a desired state, will then guide me toward what are my objective to achieve that state, and what questions shall I ask myself periodically to determine if I have reached that state. If, after I produce my set of question, I see that some of questions seem relevant, but would not directly contribute to my knowledge of my stated goal.

The following shows an example. The Intermediate Results Strategies at the bottom would then be used to develop the questions to be asked to see if you are achieving your objectives



Again, this is a framework designed to enable you to map out and clearly illustrate what you want to do. Like my college English professor told me, "you really do not know what you are thinking until you begin writing it down"

What Am I Trying to Say?

Structure – Logic – Question Flow

As I have mentioned before, I have repeatedly found the technical aspects of PDA data collection do not seem to be daunting. What I mean is that the technical aspects of the PDA hardware and the forms development software do not seem to be difficult nor limiting factors in producing a viable form design. What the teams often struggle with are in the deciphering of the structure and logic of the existing paper survey forms. The development of a PDA software form just seems to enhance any inherent problems with the existing paper forms. What I like about mobile data bases (a PDA form), is that they demand a clear logic. This can be difficult to wrestle with - clearly understanding one's form before you actually build it. But, how many of us begin pouring concrete for the foundation of a house without first thinking, discussing and producing a design to meet the requirements?


An object if inquiry (in this case a paper or PDA form) might involve 3 elements:

Structure - Logic - Question Flow

By structure, I mean how the form is literally laid out on paper - the ink, so to speak. Not only does a paper survey form (or any image of communication) have the questions typed on the paper, but there are other non-textual cues in the form design functioning as aids and guides to the form. For example, there are non-question texts such as the title of the form, instructions for using the form ("ask the question, but do not provide any answers to the respondent for the possible choices to be recorded"). Also, formatting is used to denote hierarchical levels within the form (BOLD, 16 pt. font for the major survey section, BOLD 14 pt. font for sub sections, etc.), in addition to lines of varying thicknesses, colors and a multitude of other visual cues for the form user. These form structure elements need to be understood for their intent and duplicated on a far smaller PDA display.

The logic of the form is just that. It is similar to what computer programmers employ to replicate the logic of real world thinking. For example on a paper form a question might read, "Is the woman pregnant?" If the answer is yes, move to questions 14 through 22 on pre-natal health. If the woman answers No, then continue skip questions 14 through 22 and move directly to question 23 on household income. logic is actually contained within each and every question. Each question has an "entry" rule(s) in order to govern whether the question should be asked, and each question, once it has met the entry rule(s) has an in internal set of rules that will determine validation of what can be entered as a response.
For example, we will not accept the recording of the weight of a baby of the age range of two to five years of age, of less than zero kilograms, nor greater than 100 kilograms. I am freely using hyperbole here to demonstrate my point. Once we achieve interplanetary NGO efforts, we might need to re-write the logic of our survey forms. Lastly, each question has an "exit" rule that, depending on the other rules, determines what action to take or to which question is to be shown next. Of course, the majority of questions have very simple logic and simply progress in a lineal fashion; just record Yes or No for questions 1 through 108.

The question flow is strongly related to the question logic, but it more describes the overall intent of the questionnaire. It is something like the "stand back and look and what you are trying to achieve with the entire set of questions. Sometimes one can get too involved in the logic, rules, and details of each question and forget the objective of the form. For example, "Why are we taking the time and effort to ask this mother 108 questions? I recall that we only had 4 intermediate results defined in our Results Framework which resulted in 18 variables required for program analysis?" Are why designing a coherent form that leads both the interviewer and respondent in an understandable conversation that reaches beyond a survey instrument and incorporates a human-to-human discussion relevant to the program deliverables that will improve their lives. This is actually the most important part.
So, what is this Results Framework (alternatively termed the Logic Framework), and what value can it provide in designing a survey form, whether used on paper or a PDA? The fact is, the Results Framework determines what the question will be, while ensuring that the question asked (and the response content) will provide information to determine if a program efforts are indeed producing the desired result.

The Head Teacher Interview Form

Here is how the form looked on the PDA's after we agreed on the forms' intent. The first screen identifies to the interviewer the title of the form and its purpose. Note the Save the Children branding.



The next screen, shown below, regards data privacy and reminds the interviewer to ask for the respondents permission to be interviewed.



If the respondent declines and the interviewer records "No", then the form simply moves to the end of the form and instructs the interviewer to thank the respondent for their consideration.



If the responent agrees to the interview, the first screen of questions is presented to the interviewer.



For instance, instead of having to type in the interviewers own name, they simply "tap" with the PDA's stylus (onscreen equivalent to a PC's left mouse click) on the button entitled "Lookup..." to the right of "Interviewer Name" and they are presented with a list of interviewers..



The PDA form designer can either limit the interviewer to only a selection of pre-determined interviewer names from the Lookup list, or they can allow them to type in an additional name not listed. This is a design consideration, and often you will want to restrict the selection to an exclusive list of items presented (in this case, interviewers names). But, you always need to consider what unexpected conditions that might be encountered in the field and to allow flexibility for the PDA user to add to the list on-the-fly, so to speak.myself.

There are also the flexibility in form design to offer a list of multiple selections of responses to a question ("AND" responses)



Now, let's see how the logic of the "what is your highest level of academic achievement" question displayed.



When the interviewer taps the Lookup button, they are presented with the four choices, including "Other" (notice "please specify" does not need to be shown)



If the respondent provides one of the first three responses, it is simply recorded and then moves to the next question. however if they choose "Other", then they are instead directed to a screen and asked to type in the "Other" achievement.



There is no way out of this sequence of screens - the logic must be followed, unless the interviewer makes the incorrect selection. Even then, the PDA form designer has a multitude of ways to guide, or even force, the interviewer to make the correct choices. Few of this PDA form functions are available for a paper-and-clipboard method of surveys. You can write anything on a paper form...

Monday, January 28, 2008

Save the Children - Mangochi, Malawi



This one of the largest teams that I have had for my training sessions. It is well represented from a broad cross section of the disciplines here at the Mangochi district office of Save the Children USA here in Malawi. Not only are the M&E and IT representatives here, but interested representatives from the various field programs. What a delight!



Our focus for field testing the PDA's is going the Education group, and specifically a Head Teacher PDA survey form that we will test at a couple of schools. The training was conducted in a very large conference room well suited for meetings and presentations. But, I quickly realized I would need to help the participants by dissolving the wide circle of tables in order to bring the participants in closer. This was needful not only to bring them closer together physically in order to hear me speak (I do not lack, I have heard, the ability to project my voice) and to see the training materials, but more importantly to develop the sense of a team. So, I rearranged the room (to the obvious delight to some, and dismay to others desiring to remain on the fringe) to this:




The team quickly picked up on using the PDA's, and after I quickly designed a simple form, I asked for a decision on a form that we would design and take for field testing to the program location. They selected the Head Teacher Interview form, but decided to use it not only for the head teachers (principles), but also the rank and file teachers as well.

What follows is the first few questions of this candidate form. I will focus on the 2nd question of the form: "What is your highest academic qualification?"

Save the Children
Sponsorship Primary Education Program

2007 Standard 4 Data Collection

Headteacher Interview

School:___________________________ Name of Interviewer: _____________________________

Head’s Name: ________________ __________Sex: 0__Male /1 __Female: School Code: 19/__ __

1. Standard(s) teaching _____________
2. What is your highest academic qualification?
A) JCE/ B) MSCE/ C) "A” levels/ D) Other (Specify): ___

The first few questions of this form was simple enough, only requiring brief discussion on the response codes for gender and the source of school identification codes. However, question 2 asking "What is your highest level of academic achievement?" In order to overcome my deficiency on cultural and language, I try to watch for audible and visual cues provided by the group. After asking the group what this question meant, I noticed that several dissenting sub groups emerged. I gave time for their inter- and intra- sub group discussion to occur. I not only wanted them to arrive at a consensus, but I also wanted to observe their decision dynamics, for this team in particular, and possibly cultural, as a whole progressed.

Here was the issue. Question two had, what appeared to be, four different responses to the question, "what is your highest level of academic achievement?" The response could be JCE (2A) OR MSCE (2B) OR "A” levels (2C). There was a 4th response called "Other", which I termed (2D) for discussion purposes. This last response also indicate (but I was not sure), please specify the name of the "other" academic achievement. Actually I annotate all question and survey artifacts with shortened notations. I actually utilize these notations for scripting (a simple form of programming) used by the forms design software to define and operate the logic of the form.

Since a teacher could achieve several levels of academic achievement (e.g. our Bachelors, and then later Masters, or Doctorate degrees), the question was asking for the "highest". So, the response logic was of the "OR" type. You could only respond to one, not two or more. This Other was causing the debate. Was other in addition to a response of 2A, or 2B or 2C, or was it a distinct "highest" achievement of it's own, simply unnamed in the 3 responses?

The debate went on for 1 1/2 hours. I allowed it to proceed since I did not want to influence the decision. But I was also wanting to use this issue and a cause for describing how that paper forms often hide their logic. In addition, I wanted to also point out that the issue is not with the logic of the specific question, but rather goes back to the design of the survey based on the Results Framework. At one point, rather meanly, I even asked the question of how long this questionnaire had been being used? It is amazing how paper forms can perpetuate an illogical or misunderstood indicator.

The team decided that the "Other" was just another unlisted highest academic achievement that should be captured but was not listed in the other three responses. So, 2D was an "OR" response. There was actually a lot of discussion that "Other" actually meant an "AND" response with the intent to capture any additional academic achievement.

This discussion of the minutia of a single question is certainly tedious. But, it is just one example that I began to encounter while endeavoring to have a team build a form in order to field test for value determination of PDA's to replace paper survey methods. In the very beginning I tended to overlook this close examination of the actual meaning of each of the survey questions. But, as I did, I found that our post field test reviews of the data collected tended towards lively debates of what the data meant. I then realized that the issue was neither with the data nor the PDA form, but a lack of consensus of the meaning of the question, or more accurately, what was the presumption of what the answer would provide. I saw that if there was not a clear understanding of the structure, logic and question flow, then I not only could not train a team to produce a usable PDA form, but worse yet, I would possibly perpetuate an instrument of mis-information that might directly affect peoples lives.

Nkopola Lodge - Mangochi, Malawi



We stopped by briefly at the Save the Children office here in Mangochi. After a few introductions, notably with my PDA Team Leader Francis Mabeti, Lloyd Msiska, the Information Technology (IT) manager, and Amy Richmond, the Sponsorship Program Manager.

Francis is the Monitoring and Evaluation (M&E) manager (I will describe what M&E is about later) and he and I will work together closely. Lloyd not only keeps the computer systems operational, but he also manages the transportation logistics. Amy is who I would describe as being responsible for the Mangochi Sub-Office.

Since we had been traveling for most all day, I was then taken to my lodgings during my work here. Now, I never know what kind of place I will be staying, which makes it interesting. It can be simply a guest house, hotel, or with a staff member at their private residence (my preference). I usually do not ask questions about my accommodations, but I just leave it up to the CO. They know best how to provide secure accommodations in the context of the area. All of my expenses while in country are paid for by the Westport It group of Save the Children, but the biggest issue for me is not to cause unnecessary effort or indirect expense to the CO. By this, I mean, I try to stay close to the office in order to minimize the logistics of picking me up each morning.

So, I began to be a bit disappointed in after an hour travel north we still had not reached my lodging. Later, it was explained that, as in Liliongwe, there was a huge gap between types of lodging in Mangochi. We soon turned off of the paved road onto a dirt road and arrive at the Nkopola Lodge after but a mile.



Well, was I wrong. It was beautiful. The Lodge is located right on the southwest shores of Lake Mangochi. The staff courteously greeted me and I was taken to my small room, somewhat of a small cottage with the back facing the white sandy beach of the lake. I unpacked and plugged in all of my PDA's to make sure that they had a full charge for tomorrows training session.



The next morning dawned on a stunning sight out my back sliding glass door. This was one of the beautiful places that I had been to. I called up room service and asked for my breakfast to be delivered to my room. When the young man arrived with my breakfast, I asked that he walk through and place it on the table outside so I could view the lake while dining. As I sat down, I saw that it was so beautiful that I turned to go inside and grab my PDA to take a picture. Within the few seconds time span, two monkeys took position on a low wall separating me from the adjacent room not more than four feet away. I had no idea that there were monkeys here, and they just showed up out of nowhere!



Well, these two uninvited simian breakfast guests were inching towards my breakfast even as I drew back to the table with PDA camera in hand. They removed themselves back along the wall to watch me dine. The moment that I finished and walked inside closing the glass doors, they, and two others pounced on the table and lifted the plates to lick off the contents - all without knocking over a glass or disturbing a utensil.

On the next few mornings, I began to notice that the monkeys would scurry across the cottage roof tops in parallel to the movements of the room service delivery persons. In this way, I knew when my meal was about to be delivered - when they paused on my roof. Sure enough, a knock at the door and everyone was ready to eat!