Story
Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology
Key takeaway
Generative AI is helping novices handle biology tasks more easily, raising concerns about dual-use risks as these skills could be misused.
Quick Explainer
This study examined whether large language models (LLMs) could assist novice biology students in completing a complex reverse-genetics workflow. Participants were split into an "Internet" control group and an "LLM" group that had access to frontier AI assistants. While LLM users had higher success rates on some individual tasks like cell culture, overall performance on the core workflow did not significantly improve. The findings suggest that effective LLM integration for novices may require more advanced interfaces and prompting expertise beyond what was provided in the study, highlighting a gap between in silico LLM benchmarks and real-world utility.
Deep Dive
Technical Deep Dive: Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology
Overview
This study evaluated whether large language models (LLMs) can improve the ability of novice participants to independently perform a reverse-genetics workflow in a biological laboratory setting. The researchers conducted an 8-week, investigator-blinded, two-arm randomized controlled trial with 153 participants.
Methodology
- Participants were randomized to either an Internet arm (control) or an LLM arm (intervention).
- Participants worked independently to complete five biological laboratory tasks over 39 four-hour sessions.
- The Internet arm had access to standard online resources, while the LLM arm could use frontier LLMs from Anthropic, Google DeepMind, and OpenAI.
- The primary outcome was successful completion of the core reverse genetics sequence (cell culture, molecular cloning, virus production).
- Secondary outcomes assessed success rates for individual tasks.
- Exploratory analyses looked at task progression, time-to-success, and participant perceptions of LLM utility.
Results
- LLM access did not significantly increase success across the core reverse genetics workflow. Only 5.2% in the LLM arm and 6.6% in the Internet arm met the primary outcome criteria.
- LLM participants did show higher success rates in the cell culture task (79.7% vs 62.5% in the per-protocol analysis).
- Bayesian modeling suggested a modest average uplift of 1.32x across the core task sequence, but this was uncertain (95% CrI 0.69–2.35).
- LLM participants initiated tasks in greater proportions and advanced further through the procedural sequence overall, indicating that LLMs helped overcome early barriers.
- Participants' perceptions of LLM utility decreased over time, despite similar effort and frustration levels between arms.
- Analysis of LLM transcripts suggested LLMs were more helpful for procedural tasks like cell culture than for tasks requiring sequence analysis and reagent selection.
Limitations & Uncertainties
- The study design introduced confounds that limit causal inference for individual tasks.
- The study was underpowered due to lower-than-expected baseline success rates.
- Newer biology-focused LLMs with improved capabilities were not evaluated.
- The results may not generalize to more proficient novices or users with greater LLM experience.
Implications
- The findings reveal an important gap between in silico LLM benchmarks and real-world utility for novices, underscoring the need for physical-world validation.
- Effective LLM use for novices may require more advanced interfaces and prompting expertise beyond what was provided in the study.
- Ongoing empirical evaluation is critical for adaptive, evidence-based biosecurity policy as AI capabilities and user proficiency evolve.