Not all scientific publications are equally useful to policymakers tasked with mitigating the spread and impact of diseases, especially at the start of novel epidemics and pandemics. The urgent need for actionable, evidence-based information is paramount, but the nature of preprint and peer-reviewed articles published during these times is often at odds with such goals. For example, a lack of novel results and a focus on opinions rather than evidence were common in COVID-19 publications at the start of the pandemic in 2019. This work demonstrates that it is possible to judge the utility of these articles, from a public health policy-making perspective, based on their title and/or abstracts alone, using deep Natural Language Processing (NLP) models. These models were evaluated against expert-curated COVID-19 evidence to measure their real-world feasibility at screening these scientific publications in an automated manner. Such models can be used by public health experts to triage and filter the hundreds of new daily publications on novel diseases such as COVID-19 at the start of pandemics.