What Will be the Results of My Questions?

The Results Framework

A note about the Results Framework. It is just that, a "programme logic that explains how the development objective is to be achieved, including causal relationships and underlying assumptions...” Wow! Let me give a simple and possibly error-prone example. Let's say you have suddenly had some unexpected even in your life that caused to perform some introspective review of your life's' goals. Let's say further that you lost your job and did not know how you would pay the bills and buy food. You might first think that you need to quickly get another job. But, you realize this might be an opportunity to review your life in general. You do just that, move it up a few levels and generalize. What do you want in life? Your responses might be, "I want to be happy!" Well, take it down a level. "I want to have money to buy necessities and enjoy good health."

Go further. "What will I do to enjoy good health (eat right, exercise, maintain healthy social relationships, etc.). Well, an important part might be, "how will I know that I have achieved these goals?" Well, you might define a list of measurements such as ideal weight attainment, percentage of healthy food groups in each weekly meal, defined progressive weight changes, blood pressure, cholesterol, etc. measurements over the year. As you disciplined yourself to measure these criteria of having good health, which in turn would be an element towards achieving "happiness", you would have an idea if your program of self-improvement was working...or not working. If your program was not working to the desired goals, you would either review or modify those goals (I really cannot drink another drop of organic carrot juice!), or change them altogether ("weight loss is not really producing the desired effects!).

My illustration is the questions that you would continually ask yourself would be based on your original objectives and the criteria that you defined to ensure that you were headed in the right direction to achieve that "state of happiness". The vital issue is not how healthy am I, but am I "happy".

The Results Framework is a method of defining a program's objectives, defining a strategy to achieve that objective and then to describe intermediate results that will yield indicators to tell you if you are indeed achieving the "state of happiness" for the programs beneficiaries. Likewise, in relation to survey questionnaires, it is not the question itself that is important, but what indicator was defined by the Results Framework that is described by the question so that I might monitor and evaluate whether I am achieving my strategic objectives for the programs success for the community.

So, how would the Results Framework help me build better survey forms? A clear definition of what I am trying to achieve towards a desired state, will then guide me toward what are my objective to achieve that state, and what questions shall I ask myself periodically to determine if I have reached that state. If, after I produce my set of question, I see that some of questions seem relevant, but would not directly contribute to my knowledge of my stated goal.

The following shows an example. The Intermediate Results Strategies at the bottom would then be used to develop the questions to be asked to see if you are achieving your objectives



Again, this is a framework designed to enable you to map out and clearly illustrate what you want to do. Like my college English professor told me, "you really do not know what you are thinking until you begin writing it down"

What Am I Trying to Say?

Structure – Logic – Question Flow

As I have mentioned before, I have repeatedly found the technical aspects of PDA data collection do not seem to be daunting. What I mean is that the technical aspects of the PDA hardware and the forms development software do not seem to be difficult nor limiting factors in producing a viable form design. What the teams often struggle with are in the deciphering of the structure and logic of the existing paper survey forms. The development of a PDA software form just seems to enhance any inherent problems with the existing paper forms. What I like about mobile data bases (a PDA form), is that they demand a clear logic. This can be difficult to wrestle with - clearly understanding one's form before you actually build it. But, how many of us begin pouring concrete for the foundation of a house without first thinking, discussing and producing a design to meet the requirements?


An object if inquiry (in this case a paper or PDA form) might involve 3 elements:

Structure - Logic - Question Flow

By structure, I mean how the form is literally laid out on paper - the ink, so to speak. Not only does a paper survey form (or any image of communication) have the questions typed on the paper, but there are other non-textual cues in the form design functioning as aids and guides to the form. For example, there are non-question texts such as the title of the form, instructions for using the form ("ask the question, but do not provide any answers to the respondent for the possible choices to be recorded"). Also, formatting is used to denote hierarchical levels within the form (BOLD, 16 pt. font for the major survey section, BOLD 14 pt. font for sub sections, etc.), in addition to lines of varying thicknesses, colors and a multitude of other visual cues for the form user. These form structure elements need to be understood for their intent and duplicated on a far smaller PDA display.

The logic of the form is just that. It is similar to what computer programmers employ to replicate the logic of real world thinking. For example on a paper form a question might read, "Is the woman pregnant?" If the answer is yes, move to questions 14 through 22 on pre-natal health. If the woman answers No, then continue skip questions 14 through 22 and move directly to question 23 on household income. logic is actually contained within each and every question. Each question has an "entry" rule(s) in order to govern whether the question should be asked, and each question, once it has met the entry rule(s) has an in internal set of rules that will determine validation of what can be entered as a response.
For example, we will not accept the recording of the weight of a baby of the age range of two to five years of age, of less than zero kilograms, nor greater than 100 kilograms. I am freely using hyperbole here to demonstrate my point. Once we achieve interplanetary NGO efforts, we might need to re-write the logic of our survey forms. Lastly, each question has an "exit" rule that, depending on the other rules, determines what action to take or to which question is to be shown next. Of course, the majority of questions have very simple logic and simply progress in a lineal fashion; just record Yes or No for questions 1 through 108.

The question flow is strongly related to the question logic, but it more describes the overall intent of the questionnaire. It is something like the "stand back and look and what you are trying to achieve with the entire set of questions. Sometimes one can get too involved in the logic, rules, and details of each question and forget the objective of the form. For example, "Why are we taking the time and effort to ask this mother 108 questions? I recall that we only had 4 intermediate results defined in our Results Framework which resulted in 18 variables required for program analysis?" Are why designing a coherent form that leads both the interviewer and respondent in an understandable conversation that reaches beyond a survey instrument and incorporates a human-to-human discussion relevant to the program deliverables that will improve their lives. This is actually the most important part.
So, what is this Results Framework (alternatively termed the Logic Framework), and what value can it provide in designing a survey form, whether used on paper or a PDA? The fact is, the Results Framework determines what the question will be, while ensuring that the question asked (and the response content) will provide information to determine if a program efforts are indeed producing the desired result.

The Head Teacher Interview Form

Here is how the form looked on the PDA's after we agreed on the forms' intent. The first screen identifies to the interviewer the title of the form and its purpose. Note the Save the Children branding.



The next screen, shown below, regards data privacy and reminds the interviewer to ask for the respondents permission to be interviewed.



If the respondent declines and the interviewer records "No", then the form simply moves to the end of the form and instructs the interviewer to thank the respondent for their consideration.



If the responent agrees to the interview, the first screen of questions is presented to the interviewer.



For instance, instead of having to type in the interviewers own name, they simply "tap" with the PDA's stylus (onscreen equivalent to a PC's left mouse click) on the button entitled "Lookup..." to the right of "Interviewer Name" and they are presented with a list of interviewers..



The PDA form designer can either limit the interviewer to only a selection of pre-determined interviewer names from the Lookup list, or they can allow them to type in an additional name not listed. This is a design consideration, and often you will want to restrict the selection to an exclusive list of items presented (in this case, interviewers names). But, you always need to consider what unexpected conditions that might be encountered in the field and to allow flexibility for the PDA user to add to the list on-the-fly, so to speak.myself.

There are also the flexibility in form design to offer a list of multiple selections of responses to a question ("AND" responses)



Now, let's see how the logic of the "what is your highest level of academic achievement" question displayed.



When the interviewer taps the Lookup button, they are presented with the four choices, including "Other" (notice "please specify" does not need to be shown)



If the respondent provides one of the first three responses, it is simply recorded and then moves to the next question. however if they choose "Other", then they are instead directed to a screen and asked to type in the "Other" achievement.



There is no way out of this sequence of screens - the logic must be followed, unless the interviewer makes the incorrect selection. Even then, the PDA form designer has a multitude of ways to guide, or even force, the interviewer to make the correct choices. Few of this PDA form functions are available for a paper-and-clipboard method of surveys. You can write anything on a paper form...

Save the Children - Mangochi, Malawi

This one of the largest teams that I have had for my training sessions. It is well represented from a broad cross section of the disciplines here at the Mangochi district office of Save the Children USA here in Malawi. Not only are the M&E and IT representatives here, but interested representatives from the various field programs. What a delight!



Our focus for field testing the PDA's is going the Education group, and specifically a Head Teacher PDA survey form that we will test at a couple of schools. The training was conducted in a very large conference room well suited for meetings and presentations. But, I quickly realized I would need to help the participants by dissolving the wide circle of tables in order to bring the participants in closer. This was needful not only to bring them closer together physically in order to hear me speak (I do not lack, I have heard, the ability to project my voice) and to see the training materials, but more importantly to develop the sense of a team. So, I rearranged the room (to the obvious delight to some, and dismay to others desiring to remain on the fringe) to this:




The team quickly picked up on using the PDA's, and after I quickly designed a simple form, I asked for a decision on a form that we would design and take for field testing to the program location. They selected the Head Teacher Interview form, but decided to use it not only for the head teachers (principles), but also the rank and file teachers as well.

What follows is the first few questions of this candidate form. I will focus on the 2nd question of the form: "What is your highest academic qualification?"

Save the Children
Sponsorship Primary Education Program

2007 Standard 4 Data Collection

Headteacher Interview

School:___________________________ Name of Interviewer: _____________________________

Head’s Name: ________________ __________Sex: 0__Male /1 __Female: School Code: 19/__ __

1. Standard(s) teaching _____________
2. What is your highest academic qualification?
A) JCE/ B) MSCE/ C) "A” levels/ D) Other (Specify): ___

The first few questions of this form was simple enough, only requiring brief discussion on the response codes for gender and the source of school identification codes. However, question 2 asking "What is your highest level of academic achievement?" In order to overcome my deficiency on cultural and language, I try to watch for audible and visual cues provided by the group. After asking the group what this question meant, I noticed that several dissenting sub groups emerged. I gave time for their inter- and intra- sub group discussion to occur. I not only wanted them to arrive at a consensus, but I also wanted to observe their decision dynamics, for this team in particular, and possibly cultural, as a whole progressed.

Here was the issue. Question two had, what appeared to be, four different responses to the question, "what is your highest level of academic achievement?" The response could be JCE (2A) OR MSCE (2B) OR "A” levels (2C). There was a 4th response called "Other", which I termed (2D) for discussion purposes. This last response also indicate (but I was not sure), please specify the name of the "other" academic achievement. Actually I annotate all question and survey artifacts with shortened notations. I actually utilize these notations for scripting (a simple form of programming) used by the forms design software to define and operate the logic of the form.

Since a teacher could achieve several levels of academic achievement (e.g. our Bachelors, and then later Masters, or Doctorate degrees), the question was asking for the "highest". So, the response logic was of the "OR" type. You could only respond to one, not two or more. This Other was causing the debate. Was other in addition to a response of 2A, or 2B or 2C, or was it a distinct "highest" achievement of it's own, simply unnamed in the 3 responses?

The debate went on for 1 1/2 hours. I allowed it to proceed since I did not want to influence the decision. But I was also wanting to use this issue and a cause for describing how that paper forms often hide their logic. In addition, I wanted to also point out that the issue is not with the logic of the specific question, but rather goes back to the design of the survey based on the Results Framework. At one point, rather meanly, I even asked the question of how long this questionnaire had been being used? It is amazing how paper forms can perpetuate an illogical or misunderstood indicator.

The team decided that the "Other" was just another unlisted highest academic achievement that should be captured but was not listed in the other three responses. So, 2D was an "OR" response. There was actually a lot of discussion that "Other" actually meant an "AND" response with the intent to capture any additional academic achievement.

This discussion of the minutia of a single question is certainly tedious. But, it is just one example that I began to encounter while endeavoring to have a team build a form in order to field test for value determination of PDA's to replace paper survey methods. In the very beginning I tended to overlook this close examination of the actual meaning of each of the survey questions. But, as I did, I found that our post field test reviews of the data collected tended towards lively debates of what the data meant. I then realized that the issue was neither with the data nor the PDA form, but a lack of consensus of the meaning of the question, or more accurately, what was the presumption of what the answer would provide. I saw that if there was not a clear understanding of the structure, logic and question flow, then I not only could not train a team to produce a usable PDA form, but worse yet, I would possibly perpetuate an instrument of mis-information that might directly affect peoples lives.

Nkopola Lodge - Mangochi, Malawi

We stopped by briefly at the Save the Children office here in Mangochi. After a few introductions, notably with my PDA Team Leader Francis Mabeti, Lloyd Msiska, the Information Technology (IT) manager, and Amy Richmond, the Sponsorship Program Manager.

Francis is the Monitoring and Evaluation (M&E) manager (I will describe what M&E is about later) and he and I will work together closely. Lloyd not only keeps the computer systems operational, but he also manages the transportation logistics. Amy is who I would describe as being responsible for the Mangochi Sub-Office.

Since we had been traveling for most all day, I was then taken to my lodgings during my work here. Now, I never know what kind of place I will be staying, which makes it interesting. It can be simply a guest house, hotel, or with a staff member at their private residence (my preference). I usually do not ask questions about my accommodations, but I just leave it up to the CO. They know best how to provide secure accommodations in the context of the area. All of my expenses while in country are paid for by the Westport It group of Save the Children, but the biggest issue for me is not to cause unnecessary effort or indirect expense to the CO. By this, I mean, I try to stay close to the office in order to minimize the logistics of picking me up each morning.

So, I began to be a bit disappointed in after an hour travel north we still had not reached my lodging. Later, it was explained that, as in Liliongwe, there was a huge gap between types of lodging in Mangochi. We soon turned off of the paved road onto a dirt road and arrive at the Nkopola Lodge after but a mile.



Well, was I wrong. It was beautiful. The Lodge is located right on the southwest shores of Lake Mangochi. The staff courteously greeted me and I was taken to my small room, somewhat of a small cottage with the back facing the white sandy beach of the lake. I unpacked and plugged in all of my PDA's to make sure that they had a full charge for tomorrows training session.



The next morning dawned on a stunning sight out my back sliding glass door. This was one of the beautiful places that I had been to. I called up room service and asked for my breakfast to be delivered to my room. When the young man arrived with my breakfast, I asked that he walk through and place it on the table outside so I could view the lake while dining. As I sat down, I saw that it was so beautiful that I turned to go inside and grab my PDA to take a picture. Within the few seconds time span, two monkeys took position on a low wall separating me from the adjacent room not more than four feet away. I had no idea that there were monkeys here, and they just showed up out of nowhere!



Well, these two uninvited simian breakfast guests were inching towards my breakfast even as I drew back to the table with PDA camera in hand. They removed themselves back along the wall to watch me dine. The moment that I finished and walked inside closing the glass doors, they, and two others pounced on the table and lifted the plates to lick off the contents - all without knocking over a glass or disturbing a utensil.

On the next few mornings, I began to notice that the monkeys would scurry across the cottage roof tops in parallel to the movements of the room service delivery persons. In this way, I knew when my meal was about to be delivered - when they paused on my roof. Sure enough, a knock at the door and everyone was ready to eat!

Astride the Malawi and Tanzania Border

I am about to leave for Mangochi to start the three days of training and field activities. There will be two days of training for the hardware and software and then a third day of field testing of the forms that the team designs. This will be a bit of a compressed schedule, but I have learned how to be flexible under field conditions.


I enjoyed my drive from Lilongwe to Mangochi yesterday. This very fine highway extends southeast out of Lilongwe and follows a elevated spine directly east of Tanzania. It rains intermittently here a great deal during this season. The weather is delightfully warm (thawing out from Armenia still!), but fortunately it is not hot, but it is lush and green. The views to the plains below were some of the most spectacular that I have ever seen.



At one point when I discovered that Malawi was on our left and another country which I had not visited was on our right, I asked the driver to pull over that I could enter another country without having to present my passport to an immigration official. Walking back across the highway (carefully) to our vehicle, I drew a crowd of children with my camera. They were delightful.




Later, I noticed young children standing at the side of the highway lifting up colored plastic bowls. I knew that they were selling something, so I asked the driver to pull over. He smiled and soon I found several children pushing bowls of dried ants towards me. So, I ate ants. They are a nutritious source of protein, great for on-the-road snacks for the myriad truck drivers. They have a slight slight crunchy texture with a slight hint of formic acid, and definitely do not taste like chicken!

Lilongwe, Malawi

It is Friday and the second day here in Malawi. I arrived yesterday in the early afternoon after a short stay at the Nairobi Airport in Kenya. I had been studying the security reports (I receive three from two different sources) to assess any risk to the airport. Mr. Franko, the Save the Children dedicated travel agent, informed me that there were no flight cancellations. I am not one to be overly concerned about such things. But I do like to be informed so that I can make the best decision. I learned this from Save the Children’s security unit and from a brief discussion with Paul O'Neil, whom I literally bumped into while trying to locate the coffee in the Save the Children Washington DC office.

The violence and demonstrations over the disputed elections and tribal conflicts were continuing in Nairobi and other regions to the north. The only evidence that I saw was an increased in armed guards in the airport terminal and even patrolling on the tarmac.

I was picked up by a driver from the Save the children here in the capital city of Lilongwe. Mali is a small country, but very green (during the rainy season) and full of friendly, accepting people. The country is notable for Lake Malawi, which occupies nearly 1/5 of this entire narrow land in south central Africa. This lake is often called the "year" lake, since it is approximately 365 miles long and 52 miles wide. My GPS unit showed it being at about 3,440 feet elevation, which explains in part the relatively cool weather just 13 degrees south of the equator.



Derrick, the driver, took me to the Capital Hotel. At first look, I thought to myself, "This will be comfortable". Actually, I can stand a bit more uncomfortable than this fine hotel - a little too fancy for me. In part, I saw this, because I have noted that the predisposition to being able to converse with strangers staying at a hotel is in inverse proportion to the price of the lodging. Still, I am not complaining, but the Capital Hotel will be very secure and restful before my field work commences, and the office negotiated a very fine rate for my room. I realize that it is prudent to not talk to strangers, but gee whiz, everyone has the ability to be polite when one is greeted. Maybe when I am more successful and think more highly of my position, I will be less likely to press a greeting.



On the other hand it is so easy to talk to the housekeepers, groundskeepers, etc. They are very friendly here and not at all surprised to be greeted. For instance, I walked outside the corridor from my room and noticed a man who had just set down what looked like a heavy cardboard box. I walked over to him and greeted him in English, since I have neglected to teach myself any Chichewa phrases. I could tell that he knew English probably only a bit better than my Chichewa. But, I tapped the box in front of us with my foot. He did indicate that it was full of glasses. I leaned forward slightly at my waist as he looked on, then I grasped my lower back with the palm of my hand. At this, he smiled and we both chuckled. We both understood that this was a very heavy box to lift and to carry. We indicated goodbye and we both walked away laughing. So easy....



On the other hand, when I was asked at the Save the Children office what I would like to eat for lunch, I found myself at the local eatery in the same building. I could hear the discussion on where to take me. I encounter this often as everyone sincerely wants me to feel comfortable and at home. I finally interrupted and simply asked, "Where do all of you eat for lunch?" I was told that a cafe was within this business complex just a few doors away. I replied, "That is where I want to eat also!"

Well, that was my kind of place. Reuben the Office Administrator, who accompanied me quickly pointed out that I could use a fork or a spoon for the Nsima (similar to corn grits from the American South) if I preferred. I quickly noticed that everyone in this crowded cafe were using their hands, which I am familiar with, especially since I travel to Muslim countries. After ordering our meal, Reuben directed me to a small table up front which was empty. On it were two plastic glasses filled with Western-style utensils - spoon, fork, and knife. There was no way that I was going to pick up any of those implements when the food arrived. All eyes were glancing at me time to time, especially when the food arrived. Smiles glimmered as I began eating with my right hand - the correct hand.



On our return back to the office from our dining experience I noticed a sign above a storefront advertising as an insurance agency. It stated that it Offered Protection from First Class Professionals. I know that for myself that I likely do not encounter first class professionals requiring that I obtain some level of protection from their unscrupulous advances. Rather, I probably would need protection from the mediocre grade of professionals. But, I thought that I should, in any case stop, in and ask for a list of these high-grade professionals so as to be on the alert for their subterfuge! I am not totally ignorant that my North American delivery of English communication is the only correct method in the world. Each country that uses the English language has developed their own functional linguistics of deployment. Still, one finds individual humor at the unlikeliest turns and it is best probably spared from being shared beyond one's self. You decide.


I spoke by phone with Amy Richmond who is the Program/Sponsorship manager located in the Mangochi District in the Southern Region of Malawi. We discussed my arrival in Mangochi and a few logistical details. I will have ten persons for my two day training session. This will be great! Later, I also spoke with the Country Office Director (COD) Paul MeCartney. Paul was among the most unassuming, engaging COD that I have spoken with.

Most of my day was spent with the IT Manager and Field Office Regional Technician (FORT) for the Africa Area, Kondwani Mtalimanja. On Sunday, I will travel to this area to conduct the training. We will have an opportunity to field test some forms, and then I will return late Thursday to catch a flight the following day back to the U.S.