|
|
Archive of posts tagged evaluation
While entering the building at ‘s-Gravenhekje 1A in the city centre of Amsterdam you can feel a special vibe hanging in the air. In an old warehouse packed with history, Amlab is established.
Amlab is short for Amsterdam Lab, which consists of three different organisations respectively Akvo, Text to Change and 1%CLUB. Over one and a half year ago they decided to combine their energy and creativity and found an office where they could work together. In theory these three organisations just seem to share the same office but in reality they do a lot more.
What these three organisations have in common is that they all believe in the use of technology for social change. 1%CLUB is the online marketplace that connects people with smart ideas in developing countries with people, money and knowledge around the world. Text to Change uses mobile phone technology to inform people in developing countries about all kinds of social issues. Akvo develops and runs web and mobile services and builds networks of skilled partners that can change the way development aid is allocated and reported.
The collaboration between these three is not formed by policy, donors or other official cooperation agreements. What brings them together is their common search for innovation and the energy that comes with these young organisations.
The three organisations bundled their ideas about transparency within development aid and made up a plan. With this plan there will be no more discussions about the impact of development. The impact of a project will be exposed to everyone whether it’s bad or good.
How does this work? By using simple mobile reporting tools we will share first-person stories and data with a global audience and offer crowdsourcing tools to get people involved in supporting development projects with their knowledge, time and money. We set up a ground breaking way to explain how aid impacts the lives of people at the local level. A young Kenyan will be able to share his opinion about his local health clinic or the entire health system in Kenya just by using his mobile phone. We will share these stories with a global audience and offer crowdsourcing tools to get people worldwide involved in supporting development projects with their knowledge, time and money. The progress of technology allows us to take part in this global conversation.
Yesterday the Bill and Melinda Gates foundation announced that the AmLab idea is chosen as one of the top 10 best ideas they received. After beating over more than thousand other ideas in the competition AmLab is now ready to prove what it’s worth.
Not by endless conditions written on paper but with collaboration and shared passion brought together in an inspirational old warehouse, that’s where innovation starts!
Â
Hello developers,
It’s done! Thanks to six 1%PROJECTS in Nairobi we’ve been able to test four M&E 2.0 tools: a smartphone application, participatory video, texting and blogging. In a pilot of just four days, between June 25th and June 28th, twelve highly committed participants have provided us with valuable insights into the (possible) value of M&E 2.0 and the challenges that still lie ahead.
Inspired by the thought that ‘change is good’, we’d hereby like to share our four most significant insights and changes of opinion based upon this pilot:
1. Using M&E 2.0 tools encourages enthusiasm for M&E in general.
Based upon the fact that a lot of our colleagues tend to see M&E as dull, boring and complicated, we were surprised to meet a group of motivated staff members, all eager to learn more about M&E. And the use of 2.0-oriented tools definitely increased their enthusiasm. In itself, being able to work with smartphones and flip-cams stimulated participants to carry out M&E, it actually became fun!
2. M&E 2.0 improves the appreciation and perception of organizations.
The pilot-organizations received a lot of positive feedback from actors around them, simply because of using our M&E 2.0 methodologies. In the case of Amani Kibera for instance, several community members were impressed by the fact that the organization was able to carry out a real sms campaign! Other CBO’s working in the same area approached staff members of the organization to find out how they had managed to organize this. Making use of these tools is regarded as something reserved to big NGO’s and corporations, not as a playing field for small CBO’s.
3. Availability of required M&E 2.0 technologies is still limited.
For successful implementation of the piloted M&E 2.0 tools in the long run the presence of technological hardware like smartphones, flipcams and laptops is required. This is a challenge for all of the involved organizations. The phones required for the use of texting as an M&E tool are readily available and widespread, However, this tool unfortunately has high percentages of errors and drop-outs in the process of data collection.
4. A lot of work remains to be done.
To enable organizations to keep on implementing M&E 2.0 a lot of work remains to be done increasing the usability and scalability of 2.0-tools. The front-end, but mainly the back-end of tools like a smartphone application and a texting campaign are still highly inaccessible for tech-dummies. It requires too much expertise and needs to be simplified. This will encourage project owners to continue using M&E 2.0.
As highlighted above he pilot in Nairobi has provided us with some first answers to the questions ‘Which tools (mobile, video, photo) can be used best for data collection and how?’ and ‘How can we motivate and engage project owners to use this tool?’. Now we need more input regarding the question ‘How can the input generated through these tools be visualized (e.g. a dashboard, through mapping) and which software is needed to do this?’.
Therefore, at this point we challenge you: programmers, communicators and designers, to come up with relevant solutions to this puzzle. Feel free to use our findings, analyze our raw data or ask for our opinion. We remain committed to developing M&E 2.0. At your service! 
This is a guestblog by Crosbond Boera Moseti. Research assistant on the M&E 2.0 pilot and student at Multimedia University College of Kenia.
Hey! Howdy there? Hope everyone is well. Fantastic!
Just a bite to chew on here: Would 1%CLUB partners in Kenya be able to measure the effectiveness and the in-depth performance of their projects? Well they would, but (here is the catch) not only in a M&E 1.0 kind of way. This is a rather bureaucratic and top down approach of gathering and collecting data. Project managers would be the ones who lead the project team in the fact-finding mission, supported by the project team and expensive external experts.
Accountability to the donor wouldn’t be guaranteed though. In terms of venues of disseminating this information or the tools used, you would find that there was a rather hard way of relaying the data or providing a singular source of info. So much about the disadvantages of the M&E 1.0. It expresses the need for a dynamic more advanced and user friendly way of finding these facts and measuring the impact of projects in Kenyan communities. A pilot project was hence commissioned by 1%CLUB (Marianne), EyeOpenerWorks (Henrik, Martijn) and an external eye with the name Crosbond. The aim of this pilot was to find out whether new methodology would help in finding out the ‘most significant changes’ through projects. Guess what? The project was a success story! Local communities were ecstatic about being directly involved in the impact evaluation.
In Kibera myself, facilitating the process of the NGO Amani Kibera, I got a culture shock here. I was completely mesmerized about how this NGO organize their football teams, how they were friendly, welcoming and homely and how they were able to feel the urge to emotionally, passionately and proudly involve their members in interviews, receiving conclusive and personal feedback. The work that Amani Kibera has done here!!! Can be compared to the focus that ants do have to build their castle in readiness for the tough times ahead, this was just so awesome!!)
In Tetra Park village (where the NGO Kamane operates), Henrik and I got to be invited with open arms, even with sodas. It was evident that there was an issue of language barrier but I came in handy as a translator. We walked around for a short while and we got to see the informal settlement almost similar to Kibera but sort of smaller. The fact finding, follow-up, selection process of the stories, conversations from multiple sources, trust, and learning were all real-time and with this we could be able to find multiple sources of data that were synched together.
The Most Significant Change methodology for me was how the smooth transition from the old version of evaluation to the new dynamic method of monitoring. The participating NGOs were able to relate and adapt to the system quite much faster than I expected. They conducted the process all by themselves with minimal glitches!
-Crosbond Boera Moseti
Where would 1%CLUB be without co-creation session? That’s like cappuccino without milk, or like Lois without Clark. Therefore this April 23rd a group of Monitoring & Evaluation (M&E) innovators got together in Amsterdam to co-create the future of M&E 2.0 for development cooperation. Based upon the outline presented in our previous blog representatives of Akvo, Butterfly Works, Ventanaz, Frog Design, Text to Change, and Oxfam Novib all gathered to ‘develop the ideal M&E 2.0 tool that follows all M&E 2.0 principles’. The upcoming pilot in Kenya provides a short-term opportunity to test the outcomes of this co-creation in practice.
As Joost van der Made of Frog Design puts it in the opening session of the co-creation ‘need is the mother of all inventions’. Therefore a more in-depth dialogue about the value of M&E 2.0 is started. What’s the real need? What’s the question behind the question? A brainstorm provides the insight that one specific need does not exist, but that several target groups each have their own needs and expectations from M&E 2.0: project owners, local communities, 1%members, funders, other project owners, scientists, etc. The current challenge is to invent a 2.0 tool that benefits project owners and local communities, and also has potential value for the other target groups.
A number of ideas is brought forward to respond to this challenge. A gallery of short video clips in a catchy format can for instance transfer basic info about projects and project progress to several target groups. An Open MSC process or Open Knowledge platform can build upon existing knowledge and methodology, and increase openness, crowd participation and real time communication. Participants receive incentives for uploading new stories, videos and photos. An Open data tool can connect data of 1%Project owners to other existing data. This enables project owners to learn and to put their project in a bigger picture, with ‘Hans Rosling’-like presentations and countrywide benchmarks. Using an online Dashboard project owners can moreover reflect their ambitions and planning (‘begin with the end in mind’). This will make it easier for funders/investors to compare projects and make a contribution. In addition to this dashboard Storytelling can provide a way for realtime communication using videos, geo tag photos, blogs, sms and an apps on telephones. Project are able to share updates and news quick and easy through a ‘one click update’-system.
But, as the famous T.S. Eliot already stated ‘there falls a shadow between conception and creation’: execution! Out of all the ideas brought forward, the participants consider the following characteristics of a tool most valuable an relevant to be executed in the pilot:
− An open, 2.0 translation of MSC (Most Significant Change),
− that uses different tools for data collection: mobile, texting, video, photo,
− that connects data gathered in a project and ‘open data’,
− based upon a format of questions that combines both quantitative and qualitative  data and is easy to use, and
− tested on different types of projects: (1) projects that are still designing their activities, (2) projects that have just started and (3) project that have been implemented already.
To further develop these elements into a tool three critical questions remain for the last part of the co-creation:
1. Which tools (mobile, video, photo) can be used best for data collection and how?
2. How can the input generated through these tools be visualized (e.g. a dashboard, through mapping) and which software is needed to do this?
3. How can we motivate and engage project owners to use this tool?
In three separate groups specific input related to these questions is given. And with that, the co-creation M&E 2.0 comes to an end. Thank you Mark Tiele Westra (Akvo), Ineke Aquarius (Butterfly Works), Jaap van ‘t Kruis (Ventanaz), Joost van der Made (Frog Design), Arjen Swank (Text to Change) and Peter Huisman (Oxfam Novib)!
Now, allow us to take all the input related to these three last questions back to the drawing table first, before we share the provisional answers in our next blog. This also creates the opportunity for you to become part of this co-creation process and to come up with your suggestions. Be our guest, all input is appreciated…
Keep you posted!
How do you normally feel about monitoring and evaluation (M&E)? A drag? Boring? Complicated? A ‘have to’? You wouldn’t be the only one. It’s part of the job though. But it can definitely be more simple and useful. That’s why we are excited to announce the development of something new: an M&E 2.0 toolkit, that adds a pragmatic and more fun dimension to existing methods. Because we believe you can make it work!
So what’s new? Let’s sketch some differences between current M&E and M&E 2.0 from our viewpoint:
| M&E 1.0 |
M&E 2.0 |
| Top down (led by project managers) |
Bottom up (led by the participants) |
| Experts (experts provide opinion) |
Crowd (target groups provide multiple opinions) |
| Transparency |
Open |
| One time (after intervention finished) |
Real time (continuously throughout project) |
| Accountability (to donors) |
Open source (stakeholders use the data they need) |
| Broadcasting (one source sends) |
Conversation (multiple sources are in dialogue) |
| Focus on institutions |
Focus on people |
| Control (of people and project) |
Trust (among people) |
| Justification (to donors) |
Learning (continuous learning during intervention) |
| Single data entry |
Multiple data linked to each other |
Over the past years several M&E methods & tools have been developed that fully or partly adhere to the M&E 2.0 philosophy as described above. This is not to say that 1.0 is dead, it’s just time to add new dimensions and make it more bottom-up, fuelled by the crowd, real time and more a learning experience instead of a justification method.
1%CLUB and EyeOpener Works are in the process of selecting and testing new M&E 2.0 methodology. Additional criteria for this methodology, in addition to the ones mentioned above, are simplicity/applicability, scalability, open-source and multi-disciplinarily.
Three current M&E methods & tools that in our view best match these criteria for selection of M&E 2.0 methods & tools are:
- Most significant change (MSC)
- Participatory Video (PV)
- Mobile reporting
We’ve chosen to undertake the pilot in Kenya with six 1%CLUB projects. In collaboration with other experts and relevant stakeholders we hope to come up with new methodology that aims to combine the best elements of already existing tools. The next four months will be a process of continuous reflection on our endeavours, which we’ll gladly share with you. Join our learning curve!
More to come… Keep you posted!
|
|
