Please find report and associated images in the attached report.
Executive Summary
2022 has been a year of consolidation and learning with regards to the Loop program.
We have been able to evidence a wider variety of use cases using Loop throughout this year than ever expected. We have learned about the relative importance of some aspects of the platform that we didn't recognise last year, such as the ability to report in to an independent actor to get around failing reporting systems, and the value that anonymous reporting brings to people who are vulnerable to exploitation or abuse.
We have learnt a huge amount about being a global tool that is nonetheless owned and adapted locally and how different the uses and structures are for its deployment in each country. This has helped to reinforce the decentralised locally owned structures over time. We still need to learn how to establish clear yet flexible roles and responsibilities to ensure strong ongoing partnerships. We also need to learn how to better articulate this structure and enhance the potential benefits for local actors.
What we have not been able to adequately address however is how to get collective buy-in and agreement to use Loop at a systems wide level - either at response or organisational level across countries. We do see a growing number of individuals within these larger institutions seeing the value of Loop and being an internal advocate. We also see a growing number of local organisations asking Loop to come to their country.
We remain vulnerable to limited funders and insufficient funds to improve the platform based on learning. However, as will be seen in the case studies in this report the potential for positive impact of Loop at scale is undeniable.
How was Loop used in 2022?
Numbers of Feedback on Loop
In 2022 Loop received nearly 137 stories since March. We also received over 20 Sensitive Reports.
As can be seen below, the majority of feedback was collected between September and November as organisations took Loop on as part of their own feedback mechanism.
Image 1: Stories received on Loop from January 1st to December 30th
The vast majority of the feedback was positive. Albeit stories can be tagged with more than one Type of feedback.
Image 2: Type of feedback received on Loop from January to December 2022
We see a high number of replies from organisations back to the authors across the board. This is higher than in any other country context.
Demographics of people using Loop
As can be seen in Image 3 below, the vast majority of the feedback was from women, but no stories from people self-identifying as being non binary and only one submitted from someone self-identifying as having a disability.
The majority of stories came from people between the ages of 30 and 59 but 28 were from people over 60 years of age and 2 from people between the ages of 14 and 17. We would have expected the younger age group of 18 to 29 year olds to be the largest group submitting stories but this is not the case in any country.
Image 3: Authors submitting open feedback on Loop by Age, Gender and Disability
There is still a considerable amount of people choosing not to identify their age or gender. For the first 6 months this was primarily due to feedback on Facebook being significantly quicker if you chose not to complete this information. We then adapted the process to make it just as easy to choose your gender, disability and age range as it was to skip this step if you prefer not to share this data. We expect the data on gender and age to be more robust in 2023. Preliminary results suggest that it has had an impact on the data, although not eliminating people choosing not to share their age or gender completely.
We believe some people who are living with a disability prefer not to self-identify on Loop for various reasons. Country partners suggest that this is because of cultural ideas of disability, definitions etc. Since learning this, we have added in an entire thematic filter on Protection, this includes things like: LGTBQ+, Chronically ill, Indigenous groups, legal status (refugees etc), low income families etc. The moderators can add these tags to help better understand the data.
We have learnt that organisations prioritise reporting the total numbers and percentages of people reporting into Loop, rather than the difference of the Story Type that these demographic groups are making. For example, do more women have concerns than men? Therefore, we have decided to simplify the Statistics page Age and Gender graphs to not include Story Type, as this results in duplication of author’s numbers. For example, if I submit a story that is tagged as both a Thanks and a Question my demographic data is represented in both areas. In January we will adjust the logic to only show age and gender and users can find the breakdown of the types of stories they submit elsewhere on the statistics page by using filters to identify differences in experiences among these groups.
Demographics of people submitting Sensitive Feedback
Over the course of the year we received over 20 Sensitive Stories. 100% of the Sensitive feedback was from females.
The data on Sensitive Stories about: who reports; age; gender; and disability, is still not showing on the open aggregate and anonymised statistics page. This is to ensure the anonymity of authors and we have not yet received sufficient Sensitive reports to reach the safe number to open up this data yet.
Types of Sensitive Reports being received by Loop
We can see in Image 5 below, the types of sensitive issues being reported over the course of the year. All of them are protection stories. This is different to other countries where there are also reports of fraud, corruption and misconduct.
Image 5: Sensitive Feedback Submitted on Loop from January 2022 to December 2022
In the reports all were seeking information or referrals. 8 people were referred for assistance and only 6 people received it.
The large drop is primarily due to the authors not being contactable following the initial 24 hours. We want to work on improving this data analysis further during 2023 to ensure useful actionable insights can be gained from the open data. We will also continue to improve our processes and trust building in each country and build relevant partnerships to get higher rates of referrals where possible.
Languages used on Loop
In Ukraine and Poland we are available in Ukrainian and Polish. Both Sensitive reports and Open stories were submitted in both languages, but primarily in Ukrainian. We are able to add Russian on within a week and have moderators who speak Russian. However, after consultation with local organisations, it was decided that to ensure trust in the platform, we should not yet include Russian and review this on an ongoing basis. We would love to add on Roma but are seeking partnerships in Ukraine and Poland with Roma communities to ensure this is done with them if helpful.
We see that some people said that they speak a specific language, understood the content in that language but then chose to submit their written reply in a different language. For example, some people stated they wanted to engage in the Ukrainian language but then they submitted their written feedback in Russian.
We experienced this fluidity of languages in most countries. This shows how local people need to be able to choose how they receive information but also how they choose to respond. Our systems are being improved to be able to adapt for all of these use cases. This includes, for example moderators translating and retagging languages for improved machine translation.
Which organisations are using Loop?
Loop will only be sustainable and add value to local people if the feedback is heard and responded to, so the number of people and organisations engaging with feedback on Loop is a key metric to track.
18 organisations in Poland and three from Ukraine have signed up to be notified about if a story is submitted. That includes 31 people from Poland and 6 from Ukraine. Each has a password and can more efficiently submit feedback and replies as well as be notified if any stories are posted on Loop about their organisation.
This is part of 202 organisations and 383 people worldwide, registering from over 21 countries.
This includes primarily:
1) The Loop countries of operation (Poland, Ukraine, Indonesia, Philippines, Somalia and Zambia),
2) Head offices (Holland, United Kingdom, Switzerland) as well as
3) Other countries such as 8 from Kenya, 4 from South Sudan, Iraq, Yemen etc.
Organisations represented include, some individuals from:
1) The large International Organisations such as UNICEF, UNHCR, OCHA, IFRC etc
2) The large International Non Government Organisations such as Danish Refugee Council, CARE, British Red Cross, CARITAS, Catholic Relief Services, Humanity and Inclusion etc
3) Non program related organisations such as ALNAP, Clear Global and CashCap
4) National organisations across 21 countries including our 6 operational countries
5) National authorities, specifically in the Philippines, Indonesia and Zambia
6) Local organisations
The majority of these 202 organisations are not active users of Loop but rather signed on to be notified of any stories and to respond reactively.
The vast majority of active users of Loop are local and national organisations in all of the 6 countries of operation. The uptake is significantly more rapid and positive by organisations closest to affected populations.
We see a small handful of international organisations supporting their local partners to use Loop as their organisation-wide: Feedback; Monitoring and Evaluation; Complaints and Sexual Exploitation and Abuse reporting mechanisms. These INGOs use Loop as one tool in the Capacity Building activities of local partners. Thus, supporting partners to integrate a sustainable, long term, non-project specific, tool which also helps them to meet international due diligence requirements of donors.
We do not see INGOs using Loop in any country proactively as one part of their own feedback or reporting mechanisms.
What evidence exists of Loop’s potential for impact and use?
While the above analysis is interesting to see the growth and changes over the last year the most important element is what is the Impact, or evidence of potential Impact?
The variety of use cases and local level impact as a result of using Loop has been significant in 2022. It has been more varied and interesting than expected. We have built up evidence that Loop is being used by local people:
1) to get services they need
2) to report where services are not being delivered and
3) to report safely about abuse, fraud and misconduct
We can see evidence that this information has been used by organisations to:
1) celebrate their successes and impact
2) to course-correct implementation plans
3) to respond to people with relevant and timely information
4) to use the data to inform policy and
5) to get additional funding
We do not yet have evidence of donors or policy influencers using Loop to inform their own work. Nor of researchers using trends in Loop to inform further analysis or research. This is primarily due to a lack of scale.
We have not yet been able to evidence large scale actors deploying Loop for their own purposes or collectively and as such the data is not impacting the systems level approach or thinking. We had hoped to be adopted or ‘approved’ as a tool by AAP/ CEA, PSEA or RCCE groups but this has not yet occurred in any country. Albeit receiving a lot of Gender Based Violence reports we have also not yet been adopted by key actors as a mechanism that they are rolling out as part of their action plan. We have been referenced and included on lists of possible feedback and reporting mechanisms by numerous actors including some Government Authorities.
In our annual report we outline a number of case studies from the six countries showing the above evidence. This can be found on our website https://www.talktoloop.org/blog.
Below we include the learning which is directly relevant for Poland and Ukraine:
Learning 4: We have evidence that an independent actor allows people to get around systems which are failing them.
We have multiple examples where people have used Loop to get around a system which is failing them. For example, a reporting mechanism that is not being answered, a reporting channel which is being blocked on purpose, or where no existing safe reporting mechanism is available, known or accessible for the author. This has been a common element in the Sensitive Reports to date, across all countries and for reports about Private Sector, Public Authorities, Charities and Employers.
For example: Staff who feel like the organisation they are working for is failing them and they do not know how to report safely without fear of retribution, have reported through Loop and we have been able to give information about possible options for them and organisations which might be able to offer support.
For example: Reports of trafficking rings, fraudulent police, beneficiary list manipulation etc, have all been reported to Loop. We have reported these concerns, and escalated them where appropriate. At all times protecting the identity and respecting the wishes of the survivor. Those actors to whom the individual should have been able to report, were not trusted or in some way were blocking the active management of the case. In all cases the author of the report had tried other mechanisms, had been threatened and nearly always were frightened. The anonymity of the technology and the independence of Loop was critical in these people using Loop and in the eventual conclusion of the case.
In many of the examples Loop experienced in 2022, we were able to help the survivor(s) get the support that was needed. However in some situations the system was too large or entrenched and our data was not yet able to result in the appropriate actions being taken. We see that this could change with time.
What we are learning about the role of an Independent Actor to create safer responses
Some organisations in the sector are worried about ‘bad actors’ saying falsehoods about their organisation or their work. Loop has inbuilt mechanisms to identify some such reports. We also do not share the details of the organisation or sensitive reports publicly.
Based on our learnings this year, we worry more about bad actors within the organisations using these excuses to protect their organisation over the wellbeing of local people or their staff. Fraud and abuse exists everywhere, in every organisation. To root it out people need options and workarounds to ensure issues are identified and addressed. Independent collective mechanisms will help to create a system that doesn't protect the wrong people.
At a very small scale we are seeing increased reporting of misconduct from staff after they have received training about Loop and SEAH. Thus, making them more likely to use Loop to report misconduct within their organisation. The information can be passed on anonymously, thus protecting the complainant. This makes them less of a Whistle-blower and more finding a quiet solution around perceived barriers, to an appropriate and responsive actor.
We are also seeing a pattern where those responsible organisations, actively promoting Loop as a safe reporting mechanism, are not the ones getting reports back about their own programs and staff but are rather contributing to a safer, more accountable environment. We see these same communities instead submitting sensitive, anonymised reports about other actors who may not be investing in feedback and safe reporting measures to the same degree. Thus, organisations promoting the use of Loop appear to be contributing to a safer response/ environment more broadly.
Some organisations are concerned about whether we can manage the authentication of authors and potential spam. Loop was tested on a number of occasions during 2022 and our multiple layers were able to stop misinformation and spam from infiltrating the platform - technology fixes, moderators, etc. This included duplicate feedback, election campaigning and misinformation.
What we learnt from collective aggregate open data:
Loop data shows that people feel confident to report vague, anonymised information to an independent platform and that collectively this information can be acted on. They reported the types of information and rumours that are often heard but where there are not enough facts to make an official report or complaint, but still somehow, ‘everyone knows about it’. This pattern of everyone knowing but nothing changing was identified by investigations exposing Sexual Exploitation and Abuse (SEA) cases which occur in many humanitarian contexts including in the Democratic Republic of Congo, South Sudan and Haiti, as well as the ‘Oxfam Scandal’. Investigations show that, in these examples, significant action was only taken when it was exposed in the press years later, by journalists. How many victims could be avoided if a safe, collective, independent, anonymised reporting mechanism is made available, accessible and known about?
We have also learnt that the sector is not yet seeing Loop as a viable tool to fill this gap. The reasons for this need to be explored further. We have not seen international actors promote or use Loop for SEAH reporting anywhere yet and as such have only had one SEA report all year. It is unlikely that we will see SEA reports on Loop until organisations promote it as a tool and local people are made aware of their rights and the independent, anonymous facility.
Some organisations have queried the amount of Personal Identifiable Information (PII) that we share on Loop and how this aligns with sector best practice. As a result we have asked local communities and created country specific policies on what PII can be shared, if the author gives consent, and what we should redact, even if the author gives consent for it to be shared. For example: women in Ukraine felt that Loop, redacting their name from posts, is paternalistic and condescending, suggesting they don't know how to manage their identity, especially considering they were already using Telegram and other social media platforms actively, where there is no moderation and people can contact them directly. Alternatively, survivors of abuse from Somalia or Zambia appreciated this additional level of scrutiny on what was shared online. We have thus enabled a controlled, redacting of data possible for trained moderators, based on country and story specific criteria, and securely keeping original text confidential. See our blog for more information.
We also see that organisations, representing survivors, are not yet supporting survivors to report in through Loop. During the prototyping in Zambia legal aid clinics and organisations providing shelter for survivors of sexual abuse, said that they saw the value in the aggregate data that can be collected through Loop to help them to evidence the scale and patterns of abuse that they see in their workplace. They stated that the Statistics page could help them to advocate for and influence greater accountability and increased investment in assistance services. However, we have not yet evidenced this happening and need to work more closely with key agents of change to understand what barriers they see in starting to use Loop.
Learning 6. We evidenced that real time data helps to improve data collection approaches
We know that data collection approaches are designed by people from the HQ and delivered by local populations. If these are done through systems which are paper based or offline, where all of the data is entered and analysed at the end, there is a potential to not be able to fix errors in the design process early on.
Current data collection methods involve a number of steps in the process, such as:
- an enumerator/volunteer writing a summary of what the ‘beneficiary’ said
- someone translating it
- someone inputting the data
- someone cleaning the data
- followed by data analysis by one or two people who then write the report
- the report is then reviewed, cleaned and shared with a select group of people often months after the data is collected
The use of real time data collection methods can contribute to cutting out complex steps, potential errors in the processing and make the data more actionable by more people. We have seen that the real time analysis enabled changes to the collection processes in near real time.
Other data collection tools also enable this ongoing analysis of data to inform and course correct the design of data collection processes. The important ingredients include: ongoing analysis of the data coming through while the data collection is ongoing, live updated statistical analysis to show trends as they emerge, having the authors’ qualitative input available to reinforce and add granularity to the quantitative data analysis and understanding, having the data analysis open and available for multiple users to inform their approach and activities and to make sense of the data.
People have reported that aggregated data presented in a report often leaves the author wishing for more, to better unpick and understand the possible causes or appropriate next steps. Being able to filter the data further and to swap between the resulting Qualitative and Quantitative data helps to resolve this issue.
We have a number of design improvements to the statistics pages on Loop as a result of the learning so far. These improvements, once funding permits to implement them, will help anyone to find patterns in the data more easily.
Finally we have learnt that we have a lot to learn.
We have learnt that Loop is only a shell of what it could be and that we have a lot of work to do in both the tool - technical adaptations, as well as the communications and outreach to help grow the platform and ultimately its potential to contribute to a more impactful response and to keep people safe.
While we had hoped for a larger scale uptake by now, the actual numbers can be expected after only one year of implementation at most, in each context: 1 year in the Philippines and Zambia; 8 months in Poland and Ukraine; 4 months in Indonesia and: 1 month in Somalia.
As a result, we have a long list of technical improvements we would like to make and partnerships we hope to foster.
What will it take to get to the next level?
For Loop to succeed we need to get greater buy-in and funding directly to the host organisations and Steering Groups in each country where Loop is operational. This will enable them to build stronger local partnerships, integrate research, learning and sensitisation across their national ecosystems. We would like to thank GlobalGiving for being one of the first actors to support a Loop coumtry response at the collective level.
At the global and systemic level, we would benefit from suggestions, encouragement and recommendations to use Loop by key actors - donors, AAP/ CEA/ PSEA experts etc - to help build confidence.
Attachments:
![Share on Twitter]()
![Share on Facebook]()
