How is ROOK dealing with duplication of data? How is ROOK cleaning the data?

Managing data duplication is a common challenge when integrating health data from multiple sources. ROOK Connect is designed to address these issues effectively, ensuring data accuracy and consistency. This article outlines how ROOK Connect handles data duplication and employs robust cleaning mechanisms to maintain data integrity.

Data duplication occurs when the same information is collected from multiple sources or when inconsistencies arise due to differences in data formatting across devices and platforms.
​

ROOK Connect employs a structured approach to deduplication, ensuring only the most accurate and complete data is retained. The deduplication process follows these key steps:

ROOK uses a clear hierarchy to prioritize data sources and prevent duplication of summaries and event data. Biometric devices are given the highest priority, followed by health kits. Within each category, specific devices are also prioritized. For example, Garmin data is prioritized for Physical Health, while Oura data is prioritized for Sleep.

Ensuring high-quality data requires more than just removing duplicates. ROOK Connect implements various data cleaning processes to maintain data integrity.
​

ROOK standardizes data formats and units across all integrated sources, ensuring consistency in metrics such as weight, heart rate, and timestamps. For example, distances from different devices are converted to a uniform unit (e.g., kilometers).
​

All data, including dates, weight units, and activity levels, is converted into a standardized format. This ensures seamless interpretation and analysis across platforms.
​

ROOK validates data by detecting inconsistencies, missing values, and outliers. This helps to identify potential data quality issues before further processing.
​

Data is normalized to a common scale, making it easier to compare values from different sources. This process enhances accuracy in metrics such as sleep quality and physical activity levels.
​

When multiple sources provide complementary data, ROOK intelligently combines the most complete and accurate information to generate a holistic view of the user’s health data.

ROOK delivers refined and processed data from our health categories as events and summaries. Duplicate data points for a single event are managed through ROOK's Data Duplicity feature, which prioritizes the first recorded instance. Subsequent events from different sources within a close timeframe (+/- 10 minutes) are discarded, allowing a maximum of two events within this window. Data sources directly connected to wearable devices are prioritized over health kits and SDK extractions.

The creation of summaries relies on the data available at the time of calculation. Initially, a summary is formed using the first data source received. This summary is then enhanced with extra data from other sources. Once the initial summary is dispatched, there is a 15-minute interval before an updated summary, incorporating any new data, is sent. These subsequent summaries are marked as updated versions.

ROOK Connect ensures that users receive accurate, consistent, and reliable health data by implementing a comprehensive approach to data deduplication and cleaning. By harmonizing, validating, deduplicating, and normalizing data, ROOK provides a streamlined and efficient solution for integrating health data from various sources. This improves the quality of health data analysis, enabling better decision-making for users and clients alike.

For more details on ROOK Connect’s data management capabilities, refer to our API documentation.