Panel: Care Protocols from the Future
Held on September 4, 2025, this is the first public panel in the Human in the Loop series.
Building on the comic, 'The Wandering Healer,' a speculative story about a community health worker navigating the promises and pressures of GenAI in healthcare, the panel seeks responses to the comic’s provocations, exploring how GenAI could reshape frontline care, patient relationships, and equity in India’s health systems.
The panellists were:
- Amrita Mahale, Director, Product & Innovation, ARMMAN
- Alpan Raval, Chief Scientist, AI/ML at Wadhwani AI
- Dr. Shrey Desai (MD, MPH), Trustee, SEWA Rural
- Smriti Parsheera, Independent Lawyer and Researcher, and Editor of ‘Private and Controversial: When Public Health and Privacy Meet in India’
Watch the recording of the panel and read the summary below:
Panel Summary
We opened the panel by juxtaposing AI's promise (speed, scale, consistent clinical reasoning) with the lived realities of frontline care work. Moderator, Aarushi Gupta asked the panellists to interrogate a core question: when care is mediated by algorithmic tools, what and whom are we optimising for - speed and coverage, or trust, dignity and relational care? The conversation navigated the contours of concrete field lessons and the practicalities of design, deployment and community engagement.
Alpan Raval pushed back on the simple "AI = efficiency at all costs" framing. He emphasised that in practice teams can - and should - prioritise factual accuracy, contextual fine-tuning, and human-in-the-loop design over raw throughput. He outlined concrete development practices: prompt engineering, domain-specific fine-tuning, and reinforcement learning with human feedback using on-the-ground raters. Alpan stressed evaluation as central - not off-the-shelf benchmarks but evaluations based on the actual questions frontline workers ask. He flagged technical constraints (connectivity, need for smaller local models) and a deeper risk: long-term inductive biases of models shaping human thought and behaviour, potentially producing dependence and "thinking like the model". In closing, he made a practical ask: that innovation ensures that humans who deploy AI understand and govern it for tools to remain aids and not governors of practice.
Dr Shrey Dessai brought in decades of frontline experience and grounded the panel in field realities. He described how digitisation (paper → registers → digital reminders/protocols) produced measurable health gains but also unintended effects: overloading health workers with exacting reminders, occasional data fudging to close tasks, and familial/community tensions around device use. His core message was that technology must be introduced as a ‘job aid’ not a ‘job add’ - and only after continuous listening, iteration and capacity building. Shrey highlighted practical remedies his teams used (community meetings with husbands, iterative reduction of reminder volume) and argued that humility and ongoing engagement with health workers are essential to reconcile efficiency and humane care.
Amrita Mahale presented a product and design lens, describing ARMMAN’s approach - start by asking what problem you are solving and who defines it. She detailed the design process for an ANM chatbot - choosing learning support (safer) over decision support, using Wizard-of-Oz experiments to simulate the tool, discovering voice notes and multilingual answers worked far better than text for older ANMs, and running phased rollouts with supervised follow-up before scaling. Amrita highlighted trade-offs between local contextual performance and pressure to scale: language and dialect differences created real failures when moving a Hindi bot from UP to Telangana. She said that product teams must push back on unrealistic scaling timelines when safety or contextual relevance is at stake.
Smriti Parsheera reframed the debate through standpoint theory and contextual integrity. She argued that policy and design conversations often start from bureaucratic or technocratic standpoints that privilege efficiency; and offered that reversing perspective - centring community health workers and service users - produces different conclusions about whether AI is the right tool. She cautioned mission creep and contextual integrity violations when health data collected for care gets reused in other domains (e.g., credit). On governance, Smriti urged moving beyond principle proclamations to building graded, pragmatic regulatory conversations now - because legal and operational fixes take years to materialise.
Across the panel, a consistent arc emerged: AI can improve reach and accuracy, but only if development, deployment and governance are rooted in real-world contexts and human relationships.
Technical fixes matter, but so do programme choices, sustained community engagement, and policy attention to data use and recourse. The panel closed on practical, aligned notes: insist on human-in-the-loop systems, build phased, supervised rollouts, prioritise participatory design and evaluation, and treat 'scale' as something that must not override safety or contextual fit. Humility, interdisciplinarity and iterative listening - not techno-optimism alone - are the core prescriptions the panel leaves the audience with.