Data Pulled is Inconsistent with Apple Health

[benjaminwnelson]

Hi,

I've been working through the downloaded .xml file and used your package to convert it to a dataframe, but when I look at data from specific times they do not match that data from that time on the health app on my phone. Any ideas how this could be?

Thanks again for an awesome package.

Best,
Ben

[deepankardatta]

It's difficult to actually say without examining the data. Do you have an example?

The one thing that I did notice is that Apple Health data is in quite small time periods which is then aggregated for hourly totals.

The other potential option is that it could be due to time zone conversion. You could try manipulating the data with the lubridate package, or tinker with the code in the library, to see if it makes a difference?

[benjaminwnelson]

Here's a picture from the health app vs the final heart rate output from the xml file. You can see that the sampling time is not lined up and the values are different. Note that the heart rate value is column number 5 in the excel picture below.

[deepankardatta]

Interesting. Will have to look at my own data. This might take a bit of time to work out what's going on as busy with a few other things. Are you good at R and able to have a look yourself to see where the issue is?

[benjaminwnelson]

I'm pretty busy at the moment as well. I will try to look into this when I have some time. I'll let you know if I figure anything out.

[deepankardatta]

My data input times seems to match up mostly with my Apple Health iOS display. (They are sometimes off by a minute or two on the phone, but the excel output I have shows the precision of the timing in the Apple Watch data).

I've update the library with some minor changes. Would you be able to send me a snippet of your excel file please, but in a slightly different format as below please just for the timings you sent above?

library(AppleHealthAnalysis)
library(openxlsx)
health_data <- ah_import_xml("unused/export.xml")
write.xlsx( health_data , "unused/health_data.xlsx" )

It might be that this is all just due to time zone formats, or daylight savings time, but no way to tell without looking at the data.

[benjaminwnelson]

Here it is. Yes, I'm wondering if the time zone could have to do with it. When I load the package it says,

Warning message:
In format.POSIXlt(as.POSIXlt(x, tz), format, usetz, ...) :
unknown timezone 'zone/tz/2018c.1.0/zoneinfo/America/Los_Angeles’

Here is a screenshot of the excel file in the format you specified:

[deepankardatta]

I assume I'm working at GMT. LA should be GMT-8. Could you have a look at the readings around these offset times to see if that actually matches up?

[benjaminwnelson]

Unfortunately, it looks like that isn't the issue. If I subtract or even add 8 hours the values still don't line up. I checked on a random value for a different day and the value was off by around 60.

[deepankardatta]

Not sure what to make of this. What OS and version of R are you using? (Is it an OS problem e.g. tidyverse/lubridate#615 (comment))

Might also be a timezone issue as apparently this can be OS dependent.

The other way to investigate the other way round and note the heart rates from the iOS display and see where in your excel file that pattern fits?

[deepankardatta]

Could you try the following for me to see what your computer does, after building the github version of the library please?

library(AppleHealthAnalysis)
library(lubridate)
test_time <- ymd_hms("2010-12-13 15:30:30")
tz(test_time)
health_data <- ah_import_xml("export.xml")
tz(health_data)

[deepankardatta]

Or actually even try this (feel free to skip steps above that have already been done):

library(AppleHealthAnalysis)
library(lubridate)
library(openxlsx)
Sys.timezone()
health_data <- ah_import_xml("export.xml")
tz(health_data$endDate)
health_data$endDate2 <- force_tz( health_data$endDate , tzone = "America/Los_Angeles", roll = FALSE)
tz(health_data$endDate2)
write.xlsx( health_data , "health_data_force_tz.xlsx" )

I wonder if there is an issue with what the OS timezone database is recognising what a valid timezone is. These lines should force the Los Angeles time zone onto a new column on a dataframe

[benjaminwnelson]

For your post stating, "Could you try the following for me to see what your computer does, after building the github version of the library please?

library(AppleHealthAnalysis)
library(lubridate)
test_time <- ymd_hms("2010-12-13 15:30:30")
tz(test_time)
health_data <- ah_import_xml("export.xml")
tz(health_data)"

Here is a screenshot:

[benjaminwnelson]

Just ran the second set of code you send. It creates a new column for date/time, but it matches exactly the column that is already there.

When I visually inspect values they are still off.

In addition, it's now not pulling dates past 3/4/18.