趋势解读：Towards passive heart health monitoring via smartphone camera，提升开发者接入体验

Eric S. Teasley, Product Manager, and Ming-Zher Poh, Staff Research Scientist, Google Research We present a research system that passively measures heart rate and resting heart rate via facial video captured by the front-facing camera during everyday smartphone use. Heart rate (HR), one of the cardinal vital signs , is a dynamic indicator of physiological status, influenced by everything from activity, to stress, to acute and chronic illness. Further, resting heart rate (RHR) is a key biomarker of cardiovascular health and long-term health risk. A higher RHR and increases in RHR over time are associated with major adverse cardiovascular events and all-cause mortality. Wearables, such as Fitbit devices and the Pixel Watch , have made it possible to track these health markers throughout our daily lives. However, there is room to improve their adoption, especially in low-resource environments and among those most at risk for cardiovascular disease. Smartphones present a unique opportunity to broaden access to health tracking — today, around five billion people already own a device with powerful sensors capable of monitoring their health. In 2022, we demonstrated using smartphones for on-demand HR measurement via a finger placed over the camera, and subsequent Google research considered how the signal detected during that measurement could help predict cardiovascular disease. In “ Passive Heart Rate Monitoring During Smartphone Use in Everyday Life ”, published in Nature , we introduce a research system (PHRM) that enables tracking of HR and RHR in the background during everyday smartphone use. PHRM leverages the front-facing camera to capture video of the user’s face in the seconds after face unlock events. It then applies deep learning to estimate HR with a mean absolute percentage error (MAPE) < 10% compared to electrocardiogram -derived ground truth, meeting industry accuracy standards for people of all skin tones. Finally, the system integrates HR measurements throughout the day into an estimate of daily RHR that matches the accuracy of wearables, with a mean absolute error (MAE) of < 5 beats per minute (bpm) compared to a wearable tracker. With our publication, we release the largest and most diverse dataset of smartphone videos publicly available for research along with a pre-trained “PHRM-mini” model. Qualified researchers can apply for access. Like wearables, pulse oximeters , and our previous work , PHRM measures HR via photoplethysmography (PPG), i.e., by sensing the fluctuation in how light interacts with the skin each time blood pulses through it. We developed an on-device software pipeline that processes 8-second facial video clips and uses computationally-efficient temporal shift convolutional neural networks to predict HR along with a confidence score. The pipeline further aggregates HR predictions over the day and leverages confidence scores and Kalman filtering to estimate a daily RHR. PHRM’s pipeline for estimating HR and daily RHR from clips of a user’s face. While computer vision models for such “remote” PPG (rPPG) have existed for two decades, previous work involved smaller studies under controlled conditions, limiting generalizability. Additionally, previous studies vastly underrepresented people with dark skin, in whom melanin makes the PPG signal more challenging for cameras to detect. Only recently have researchers investigated rPPG model performance on dark-skinned study participants more thoroughly, finding significantly lower accuracy — a trajectory similar to what has occurred for pulse oximeters and other PPG-based technologies. The concerns about pulse oximeters spurred the FDA to draft guidance to ensure diverse skin tone representation in validation studies. Thus far, there is a lack of studies of rPPG that achieve similar standards. We developed PHRM using over 350,000 video clips from nearly 700 diverse consented research participants in both laboratory and real-world settings, and we de