{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Reading\n", "\n", "In principle, Niimpy can deal with any files of any format - you only need to convert them to a DataFrame. Still, it is very useful to have some common formats, so we present two standard formats with default readers:\n", "\n", "* **CSV files** are very standard and normal to create and understand, but in order to deal with them everything must be loaded into memory.\n", "* **Google TakeOut** provides a large selection of data in different formats. We provide readers most commonly used data types.\n", "* **MHealth** is a common format for health data.\n", "* **sqlite3 databases**, which requires sqlite3 to read, provides more power for filtering and automatic processing without reading everything into memory.\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## DataFrame format (in-memory)\n", "\n", "In-memory, data is stored in a [pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html). This is basically a normal dataframe. There are some standardized columns (see the [schema](/about/schema.html)) and the index is a DatetimeIndex." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## CSV files\n", "\n", "CSV files should have a header that lists the column names and generally be readable by `pandas.read_csv`.\n", "\n", "Reading these can be done with `niimpy.read_csv`:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "import niimpy \n", "import niimpy.config as config\n", "\n", "# Read the battery data\n", "df= niimpy.read_csv(config.MULTIUSER_AWARE_BATTERY_PATH, tz='Europe/Helsinki')" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## sqlite3 databases\n", "\n", "For the purposes of niimpy, sqlite3 databases can generally be seen as supercharged CSV files.\n", "\n", "A single database file could contain multiple datasets within it, thus when reading them a **table name** must be specified.\n", "\n", "One reads the entire database into memory using `sqlite.read_sqlite`:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# Read the sqlite3 data\n", "df= niimpy.read_sqlite(config.SQLITE_SINGLEUSER_PATH, table=\"AwareScreen\", tz='Europe/Helsinki')" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "You can list the tables within a database using `niimpy.reading.read.read_sqlite_tables`:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "{'AwareScreen'}" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "niimpy.reading.read.read_sqlite_tables(config.SQLITE_SINGLEUSER_PATH)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "sqlite3 files are highly recommended as a data storage format, since many common exploration options can be done within the database itself without reading the whole data into memory or writing an iterator. However, the interface is more difficult to use. Niimpy (before 2021-07) used this as its primary interface, but since then this interface has been de-emphasized. You can read more in [the database section](/database.html), but this is only recommended if you need efficiency when using massive amounts of data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Note of incorrect encoding\n", "\n", "Sometimes a sqlite3 database has incorrect metadata for character encoding. This is a common issue with languages that cannot be encoded with `utf-8`, such as Finnish. When this happens, use the function `niimpy.utils.set_encoding()`.\n", "\n", "```Python\n", "import niimpy\n", "df = niimpy.read_sqlite(\n", " config.SQLITE_SINGLEUSER_PATH,\n", " table=\"AwareScreen\",\n", " tz='Europe/Helsinki'\n", ")\n", "df = niimpy.utils.set_encoding(\n", " df,\n", " to_encoding = 'utf-8',\n", " from_encoding = 'iso-8859-1'\n", ")\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Google TakeOut\n", "\n", "Google takeout contains a many different types of data and new types are added as Google creates services or changes data storage methods. Readers are currently available for location data, emails, and activity data from the fit app. For other data types, the user needs to manually convert them into a Niimpy compatible Pandas DataFrame." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# Data downloaded from Google Takeout is compressed as a zip archive to conserve disk space. To \n", "# demonstrate reading for the zip file, we will first compress our example data into the zip format.\n", "import zipfile\n", "test_zip = zipfile.ZipFile(\"test.zip\", mode=\"w\")\n", "\n", "for dirpath,dirs,files in os.walk(config.GOOGLE_TAKEOUT_DIR):\n", " for f in files:\n", " filename = os.path.join(dirpath, f)\n", " filename_in_zip = filename.replace(config.GOOGLE_TAKEOUT_DIR, \"\")\n", " test_zip.write(filename, filename_in_zip)\n", "\n", "test_zip.close()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Location" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
accuracysourcedeviceplaceidformfactoraltitudeverticalaccuracyplatformtypeservertimestampdevicetimestampbatterychargingvelocityheadinglatitudelongitudeinferred_latitudeinferred_longitudeactivity_typeactivity_inference_confidenceuser
timestamp
2016-08-12 19:29:43.821000+00:0025WIFI-577680260NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN35.997488-78.922194NaNNaNNaNNaNf1066868-2e1d-11ef-abd1-b0dcef010c43
2016-08-12 19:30:49.531000+00:0021WIFI-577680260NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN35.997559-78.922504NaNNaNSTILL62.0f1066868-2e1d-11ef-abd1-b0dcef010c43
2016-08-12 19:31:49.531000+00:0021WIFI-577680260ChIJS_5Nmuz1jUYRGYf3QiiZco4PHONENaNNaNNaNNaNNaNNaNNaNNaN35.997559-78.92250460.18713524.824478STILL62.0f1066868-2e1d-11ef-abd1-b0dcef010c43
2016-08-12 21:15:55.295000+00:001500CELL-577680260NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN36.000870-78.923343NaNNaNON_FOOT54.0f1066868-2e1d-11ef-abd1-b0dcef010c43
2016-08-12 21:16:33+00:008GPS-577680260NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN35.997250-78.923989NaNNaNNaNNaNf1066868-2e1d-11ef-abd1-b0dcef010c43
2016-08-12 21:16:48+00:003GPS-577680260NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN35.997236-78.924124NaNNaNNaNNaNf1066868-2e1d-11ef-abd1-b0dcef010c43
2023-11-21 11:29:21.730000+00:0013WIFI1832214273ChIJw1WKQev1jUYRCdZmYR-HCiIPHONE28.02.0ANDROID2023-11-21T11:29:24.747Z2023-11-21T11:29:24.350ZFalse60.18681824.82128860.18681624.821288NaNNaNf1066868-2e1d-11ef-abd1-b0dcef010c43
\n", "
" ], "text/plain": [ " accuracy source device \\\n", "timestamp \n", "2016-08-12 19:29:43.821000+00:00 25 WIFI -577680260 \n", "2016-08-12 19:30:49.531000+00:00 21 WIFI -577680260 \n", "2016-08-12 19:31:49.531000+00:00 21 WIFI -577680260 \n", "2016-08-12 21:15:55.295000+00:00 1500 CELL -577680260 \n", "2016-08-12 21:16:33+00:00 8 GPS -577680260 \n", "2016-08-12 21:16:48+00:00 3 GPS -577680260 \n", "2023-11-21 11:29:21.730000+00:00 13 WIFI 1832214273 \n", "\n", " placeid formfactor \\\n", "timestamp \n", "2016-08-12 19:29:43.821000+00:00 NaN NaN \n", "2016-08-12 19:30:49.531000+00:00 NaN NaN \n", "2016-08-12 19:31:49.531000+00:00 ChIJS_5Nmuz1jUYRGYf3QiiZco4 PHONE \n", "2016-08-12 21:15:55.295000+00:00 NaN NaN \n", "2016-08-12 21:16:33+00:00 NaN NaN \n", "2016-08-12 21:16:48+00:00 NaN NaN \n", "2023-11-21 11:29:21.730000+00:00 ChIJw1WKQev1jUYRCdZmYR-HCiI PHONE \n", "\n", " altitude verticalaccuracy platformtype \\\n", "timestamp \n", "2016-08-12 19:29:43.821000+00:00 NaN NaN NaN \n", "2016-08-12 19:30:49.531000+00:00 NaN NaN NaN \n", "2016-08-12 19:31:49.531000+00:00 NaN NaN NaN \n", "2016-08-12 21:15:55.295000+00:00 NaN NaN NaN \n", "2016-08-12 21:16:33+00:00 NaN NaN NaN \n", "2016-08-12 21:16:48+00:00 NaN NaN NaN \n", "2023-11-21 11:29:21.730000+00:00 28.0 2.0 ANDROID \n", "\n", " servertimestamp \\\n", "timestamp \n", "2016-08-12 19:29:43.821000+00:00 NaN \n", "2016-08-12 19:30:49.531000+00:00 NaN \n", "2016-08-12 19:31:49.531000+00:00 NaN \n", "2016-08-12 21:15:55.295000+00:00 NaN \n", "2016-08-12 21:16:33+00:00 NaN \n", "2016-08-12 21:16:48+00:00 NaN \n", "2023-11-21 11:29:21.730000+00:00 2023-11-21T11:29:24.747Z \n", "\n", " devicetimestamp batterycharging \\\n", "timestamp \n", "2016-08-12 19:29:43.821000+00:00 NaN NaN \n", "2016-08-12 19:30:49.531000+00:00 NaN NaN \n", "2016-08-12 19:31:49.531000+00:00 NaN NaN \n", "2016-08-12 21:15:55.295000+00:00 NaN NaN \n", "2016-08-12 21:16:33+00:00 NaN NaN \n", "2016-08-12 21:16:48+00:00 NaN NaN \n", "2023-11-21 11:29:21.730000+00:00 2023-11-21T11:29:24.350Z False \n", "\n", " velocity heading latitude longitude \\\n", "timestamp \n", "2016-08-12 19:29:43.821000+00:00 NaN NaN 35.997488 -78.922194 \n", "2016-08-12 19:30:49.531000+00:00 NaN NaN 35.997559 -78.922504 \n", "2016-08-12 19:31:49.531000+00:00 NaN NaN 35.997559 -78.922504 \n", "2016-08-12 21:15:55.295000+00:00 NaN NaN 36.000870 -78.923343 \n", "2016-08-12 21:16:33+00:00 NaN NaN 35.997250 -78.923989 \n", "2016-08-12 21:16:48+00:00 NaN NaN 35.997236 -78.924124 \n", "2023-11-21 11:29:21.730000+00:00 60.186818 24.821288 \n", "\n", " inferred_latitude inferred_longitude \\\n", "timestamp \n", "2016-08-12 19:29:43.821000+00:00 NaN NaN \n", "2016-08-12 19:30:49.531000+00:00 NaN NaN \n", "2016-08-12 19:31:49.531000+00:00 60.187135 24.824478 \n", "2016-08-12 21:15:55.295000+00:00 NaN NaN \n", "2016-08-12 21:16:33+00:00 NaN NaN \n", "2016-08-12 21:16:48+00:00 NaN NaN \n", "2023-11-21 11:29:21.730000+00:00 60.186816 24.821288 \n", "\n", " activity_type activity_inference_confidence \\\n", "timestamp \n", "2016-08-12 19:29:43.821000+00:00 NaN NaN \n", "2016-08-12 19:30:49.531000+00:00 STILL 62.0 \n", "2016-08-12 19:31:49.531000+00:00 STILL 62.0 \n", "2016-08-12 21:15:55.295000+00:00 ON_FOOT 54.0 \n", "2016-08-12 21:16:33+00:00 NaN NaN \n", "2016-08-12 21:16:48+00:00 NaN NaN \n", "2023-11-21 11:29:21.730000+00:00 NaN NaN \n", "\n", " user \n", "timestamp \n", "2016-08-12 19:29:43.821000+00:00 f1066868-2e1d-11ef-abd1-b0dcef010c43 \n", "2016-08-12 19:30:49.531000+00:00 f1066868-2e1d-11ef-abd1-b0dcef010c43 \n", "2016-08-12 19:31:49.531000+00:00 f1066868-2e1d-11ef-abd1-b0dcef010c43 \n", "2016-08-12 21:15:55.295000+00:00 f1066868-2e1d-11ef-abd1-b0dcef010c43 \n", "2016-08-12 21:16:33+00:00 f1066868-2e1d-11ef-abd1-b0dcef010c43 \n", "2016-08-12 21:16:48+00:00 f1066868-2e1d-11ef-abd1-b0dcef010c43 \n", "2023-11-21 11:29:21.730000+00:00 f1066868-2e1d-11ef-abd1-b0dcef010c43 " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Next we read location data from the zip file.\n", "import niimpy\n", "import niimpy.config as config\n", "import niimpy.preprocessing.location as nilo\n", "\n", "data = niimpy.reading.google_takeout.location_history(\"test.zip\")\n", "data = nilo.filter_location(\n", " data,\n", " latitude_column = \"latitude\",\n", " longitude_column = \"longitude\",\n", " remove_disabled=False, remove_network=False, remove_zeros=True\n", ")\n", "data\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Activity\n", "\n", "Activity data is read similarly. The data contains many columns with missing data, so in order to use the step count data, for example, we must set the NaN values to 0." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
calories_(kcal)step_count
timestamp
2023-11-20 00:00:00+02:001752.2500270.000000
2023-11-21 00:00:00+02:001996.45674689.900002
\n", "
" ], "text/plain": [ " calories_(kcal) step_count\n", "timestamp \n", "2023-11-20 00:00:00+02:00 1752.250027 0.000000\n", "2023-11-21 00:00:00+02:00 1996.456746 89.900002" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = niimpy.reading.google_takeout.activity(\"test.zip\")\n", "data.loc[data[\"step_count\"].isna(), \"step_count\"] = 0\n", "data[[\"calories_(kcal)\", \"step_count\"]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Google Fit Takeout data can contain more detailed information. To access this data, we first list all data types stored in Google Fit data." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
filenamederivedcontentsourcesource type
0raw_com.google.step_count.delta_fi.polar.polar...rawstep_count.deltafi.test.teststep count
1derived_com.google.step_count.delta_com.google...derivedstep_count.deltacom.google.android.fittop_level
2derived_com.google.respiratory_rate_com.google...derivedrespiratory_ratecom.google.android.gmsmerge_respiratory_rate
3raw_com.google.sleep.segment_com.urbandroid.sl...rawsleep.segmentcom.urbandroid.sleepsaa-generic
4raw_com.google.heart_rate.summary_fi.polar.pol...rawheart_rate.summaryfi.test.testtest
5raw_com.google.calories.expended_fi.polar.pola...rawcalories.expendedfi.manual.manualtest
6derived_com.google.sleep.segment_com.google.an...derivedsleep.segmentcom.google.android.gmsmerged
7raw_com.google.nutrition_com.fourtechnologies....rawnutritioncom.manual.test
8raw_com.google.heart_rate.bpm_nl.appyhapps.hea...rawheart_rate.bpmfi.test.testTest - heart rate
9raw_com.google.sleep.segment_nl.appyhapps.heal...rawsleep.segmentnl.test.healthsyncHealthSync_sleep_by_Health_Sync_1715724879000
10raw_com.google.heart_minutes_com.google.androi...rawheart_minutescom.google.android.apps.fitnessuser_input
\n", "
" ], "text/plain": [ " filename derived \\\n", "0 raw_com.google.step_count.delta_fi.polar.polar... raw \n", "1 derived_com.google.step_count.delta_com.google... derived \n", "2 derived_com.google.respiratory_rate_com.google... derived \n", "3 raw_com.google.sleep.segment_com.urbandroid.sl... raw \n", "4 raw_com.google.heart_rate.summary_fi.polar.pol... raw \n", "5 raw_com.google.calories.expended_fi.polar.pola... raw \n", "6 derived_com.google.sleep.segment_com.google.an... derived \n", "7 raw_com.google.nutrition_com.fourtechnologies.... raw \n", "8 raw_com.google.heart_rate.bpm_nl.appyhapps.hea... raw \n", "9 raw_com.google.sleep.segment_nl.appyhapps.heal... raw \n", "10 raw_com.google.heart_minutes_com.google.androi... raw \n", "\n", " content source \\\n", "0 step_count.delta fi.test.test \n", "1 step_count.delta com.google.android.fit \n", "2 respiratory_rate com.google.android.gms \n", "3 sleep.segment com.urbandroid.sleep \n", "4 heart_rate.summary fi.test.testtest \n", "5 calories.expended fi.manual.manualtest \n", "6 sleep.segment com.google.android.gms \n", "7 nutrition com.manual.test \n", "8 heart_rate.bpm fi.test.test \n", "9 sleep.segment nl.test.healthsync \n", "10 heart_minutes com.google.android.apps.fitness \n", "\n", " source type \n", "0 step count \n", "1 top_level \n", "2 merge_respiratory_rate \n", "3 saa-generic \n", "4 \n", "5 \n", "6 merged \n", "7 \n", "8 Test - heart rate \n", "9 HealthSync_sleep_by_Health_Sync_1715724879000 \n", "10 user_input " ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "niimpy.reading.google_takeout.fit_list_data(\"test.zip\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have two types of step count data. The original raw data is in \"raw_com.google.step_count.delta_fi.polar.polar.json\". You can read this data using `fit_read_data`." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
measurement_indexidvalueend_timemodified_timedatatype
timestamp
2024-04-30 17:59:58.9990082024-04-30 19:17:002024-05-01 03:15:57.130step_count.delta
2024-04-30 17:59:58.9991082024-04-30 19:17:302024-04-30 20:02:14.506step_count.delta
\n", "
" ], "text/plain": [ " measurement_index id value end_time \\\n", "timestamp \n", "2024-04-30 17:59:58.999 0 0 8 2024-04-30 19:17:00 \n", "2024-04-30 17:59:58.999 1 0 8 2024-04-30 19:17:30 \n", "\n", " modified_time datatype \n", "timestamp \n", "2024-04-30 17:59:58.999 2024-05-01 03:15:57.130 step_count.delta \n", "2024-04-30 17:59:58.999 2024-04-30 20:02:14.506 step_count.delta " ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "niimpy.reading.google_takeout.fit_read_data(\"test.zip\", \"raw_com.google.step_count.delta_fi.polar.polar.json\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Email and Chat\n", "\n", "The `google_takeout.email_activity` and `google_takeout.chat` function will read and process all emails in the GMail mailbox and all Google chat messages respectively. They return a dataframe containing metadata and statistics of each message. Email addresses, email IDs and names are replaced by numerical indexes.\n", "\n", "The email files can be large and processing them could take some time. You can also include sentiment analysis of each email using the `sentiment` parameter. For this, we recommend using a system with a GPU." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/u/24/rantahj1/unix/src/niimpy/niimpy/reading/google_takeout.py:466: UserWarning: Could not parse message timestamp: 2023-12-15 12:19:43+00:00\n", " warnings.warn(f\"Could not parse message timestamp: {received}\")\n", "/u/24/rantahj1/unix/src/niimpy/niimpy/reading/google_takeout.py:480: UserWarning: Failed to format received time: Sat, 15 DeNot a timec 2023 12:19:43 0000\n", " warnings.warn(f\"Failed to format received time: {received}\")\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
receivedfromtoccbccmessage_idin_reply_tocharacter_countword_countmessage_typeuser
timestamp
2023-12-15 12:19:43+00:00NaT0[2][][]2<NA>336outgoingf10e4d26-2e1d-11ef-abd1-b0dcef010c43
2023-12-15 12:29:43+00:00NaT0[6, 2][][]1<NA>316outgoingf10e4d26-2e1d-11ef-abd1-b0dcef010c43
2023-12-15 12:29:43+00:00NaT0[6, 2][][]1<NA>285outgoingf10e4d26-2e1d-11ef-abd1-b0dcef010c43
2023-12-15 12:39:43+00:002023-12-15 12:19:43+00:006[0][4][4]32305incomingf10e4d26-2e1d-11ef-abd1-b0dcef010c43
2023-12-15 12:39:43+00:00Sat, 15 DeNot a timec 2023 12:19:43 00006[0][4][4]32517incomingf10e4d26-2e1d-11ef-abd1-b0dcef010c43
\n", "
" ], "text/plain": [ " received from \\\n", "timestamp \n", "2023-12-15 12:19:43+00:00 NaT 0 \n", "2023-12-15 12:29:43+00:00 NaT 0 \n", "2023-12-15 12:29:43+00:00 NaT 0 \n", "2023-12-15 12:39:43+00:00 2023-12-15 12:19:43+00:00 6 \n", "2023-12-15 12:39:43+00:00 Sat, 15 DeNot a timec 2023 12:19:43 0000 6 \n", "\n", " to cc bcc message_id in_reply_to \\\n", "timestamp \n", "2023-12-15 12:19:43+00:00 [2] [] [] 2 \n", "2023-12-15 12:29:43+00:00 [6, 2] [] [] 1 \n", "2023-12-15 12:29:43+00:00 [6, 2] [] [] 1 \n", "2023-12-15 12:39:43+00:00 [0] [4] [4] 3 2 \n", "2023-12-15 12:39:43+00:00 [0] [4] [4] 3 2 \n", "\n", " character_count word_count message_type \\\n", "timestamp \n", "2023-12-15 12:19:43+00:00 33 6 outgoing \n", "2023-12-15 12:29:43+00:00 31 6 outgoing \n", "2023-12-15 12:29:43+00:00 28 5 outgoing \n", "2023-12-15 12:39:43+00:00 30 5 incoming \n", "2023-12-15 12:39:43+00:00 51 7 incoming \n", "\n", " user \n", "timestamp \n", "2023-12-15 12:19:43+00:00 f10e4d26-2e1d-11ef-abd1-b0dcef010c43 \n", "2023-12-15 12:29:43+00:00 f10e4d26-2e1d-11ef-abd1-b0dcef010c43 \n", "2023-12-15 12:29:43+00:00 f10e4d26-2e1d-11ef-abd1-b0dcef010c43 \n", "2023-12-15 12:39:43+00:00 f10e4d26-2e1d-11ef-abd1-b0dcef010c43 \n", "2023-12-15 12:39:43+00:00 f10e4d26-2e1d-11ef-abd1-b0dcef010c43 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "niimpy.reading.google_takeout.email_activity(\"test.zip\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Google chat data is read similarly.\n", "\n", "Additionally, you can run sentiment analysis on email and chat messages using the `sentiment` option. To run sentiment analysis, you need to install the optional dependency by running.\n", "\n", "```bash\n", "pip install niimpy[sentiment]\n", "```" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/u/24/rantahj1/unix/src/niimpy/niimpy/reading/google_takeout.py:648: UserWarning: Multiple user emails found. Using the first one.\n", " warnings.warn(\"Multiple user emails found. Using the first one.\")\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " 0%| | 0/4 [00:01\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
topic_idmessage_idchat_groupcreator_namecreator_emailcreator_user_typecharacter_countword_countusermessage_typesentimentsentiment_score
timestamp
2024-01-30 13:27:33+00:00iDImYGRudHk9guW_0AAAAE/iDImYGRudHk/iDImYGRudHk000Human51f1144500-2e1d-11ef-abd1-b0dcef010c43outgoingnone0.000000
2024-01-30 13:29:10+00:00cVEoT9zu63M9guW_0AAAAE/cVEoT9zu63M/cVEoT9zu63M011Human51f1144500-2e1d-11ef-abd1-b0dcef010c43incomingnone0.000000
2024-01-30 13:29:17+00:00qEfkUgUvX809guW_0AAAAE/qEfkUgUvX80/qEfkUgUvX80011Human113f1144500-2e1d-11ef-abd1-b0dcef010c43incomingpositive0.535310
2024-01-30 13:29:17+00:00qEfkUgUvX809guW_0AAAAE/qEfkUgUvX80/qEfkUgUvX80000Human225f1144500-2e1d-11ef-abd1-b0dcef010c43outgoingpositive0.912528
\n", "" ], "text/plain": [ " topic_id message_id \\\n", "timestamp \n", "2024-01-30 13:27:33+00:00 iDImYGRudHk 9guW_0AAAAE/iDImYGRudHk/iDImYGRudHk \n", "2024-01-30 13:29:10+00:00 cVEoT9zu63M 9guW_0AAAAE/cVEoT9zu63M/cVEoT9zu63M \n", "2024-01-30 13:29:17+00:00 qEfkUgUvX80 9guW_0AAAAE/qEfkUgUvX80/qEfkUgUvX80 \n", "2024-01-30 13:29:17+00:00 qEfkUgUvX80 9guW_0AAAAE/qEfkUgUvX80/qEfkUgUvX80 \n", "\n", " chat_group creator_name creator_email \\\n", "timestamp \n", "2024-01-30 13:27:33+00:00 0 0 0 \n", "2024-01-30 13:29:10+00:00 0 1 1 \n", "2024-01-30 13:29:17+00:00 0 1 1 \n", "2024-01-30 13:29:17+00:00 0 0 0 \n", "\n", " creator_user_type character_count word_count \\\n", "timestamp \n", "2024-01-30 13:27:33+00:00 Human 5 1 \n", "2024-01-30 13:29:10+00:00 Human 5 1 \n", "2024-01-30 13:29:17+00:00 Human 11 3 \n", "2024-01-30 13:29:17+00:00 Human 22 5 \n", "\n", " user message_type \\\n", "timestamp \n", "2024-01-30 13:27:33+00:00 f1144500-2e1d-11ef-abd1-b0dcef010c43 outgoing \n", "2024-01-30 13:29:10+00:00 f1144500-2e1d-11ef-abd1-b0dcef010c43 incoming \n", "2024-01-30 13:29:17+00:00 f1144500-2e1d-11ef-abd1-b0dcef010c43 incoming \n", "2024-01-30 13:29:17+00:00 f1144500-2e1d-11ef-abd1-b0dcef010c43 outgoing \n", "\n", " sentiment sentiment_score \n", "timestamp \n", "2024-01-30 13:27:33+00:00 none 0.000000 \n", "2024-01-30 13:29:10+00:00 none 0.000000 \n", "2024-01-30 13:29:17+00:00 positive 0.535310 \n", "2024-01-30 13:29:17+00:00 positive 0.912528 " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "niimpy.reading.google_takeout.chat(\"test.zip\", sentiment=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Youtube Activity\n", "\n", "Finally, we have a reader for extracting Youtube watch history data. We do not, by default, return video identifiers, but replace them with numerical IDs. The only available information then is the recorded time, which corresponds to video start time.\n", "\n", "Importantly, we have no information on how long the user watched a given video, as this is not stored in the TakeOut data. You can deduce whether the user has rewatched a given video, watched multiple videos in a row, or started another video quickly without finishing the previous one." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
video_titlechannel_titleuser
timestamp
2024-02-13 08:36:49+02:0000f1f6ae72-2e1d-11ef-abd1-b0dcef010c43
2024-02-13 08:36:05+02:0011f1f6ae72-2e1d-11ef-abd1-b0dcef010c43
2024-02-13 08:35:38+02:0022f1f6ae72-2e1d-11ef-abd1-b0dcef010c43
2024-02-13 08:35:03+02:0000f1f6ae72-2e1d-11ef-abd1-b0dcef010c43
\n", "
" ], "text/plain": [ " video_title channel_title \\\n", "timestamp \n", "2024-02-13 08:36:49+02:00 0 0 \n", "2024-02-13 08:36:05+02:00 1 1 \n", "2024-02-13 08:35:38+02:00 2 2 \n", "2024-02-13 08:35:03+02:00 0 0 \n", "\n", " user \n", "timestamp \n", "2024-02-13 08:36:49+02:00 f1f6ae72-2e1d-11ef-abd1-b0dcef010c43 \n", "2024-02-13 08:36:05+02:00 f1f6ae72-2e1d-11ef-abd1-b0dcef010c43 \n", "2024-02-13 08:35:38+02:00 f1f6ae72-2e1d-11ef-abd1-b0dcef010c43 \n", "2024-02-13 08:35:03+02:00 f1f6ae72-2e1d-11ef-abd1-b0dcef010c43 " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "niimpy.reading.google_takeout.youtube_watch_history(\"test.zip\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since Google takeout may provide the mailbox as a single uncompressed file, it is also possible to provide it's file path directly." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/u/24/rantahj1/unix/src/niimpy/niimpy/reading/google_takeout.py:466: UserWarning: Could not parse message timestamp: 2023-12-15 12:19:43+00:00\n", " warnings.warn(f\"Could not parse message timestamp: {received}\")\n", "/u/24/rantahj1/unix/src/niimpy/niimpy/reading/google_takeout.py:480: UserWarning: Failed to format received time: Sat, 15 DeNot a timec 2023 12:19:43 0000\n", " warnings.warn(f\"Failed to format received time: {received}\")\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running sentiment analysis on 5 messages.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " 0%| | 0/5 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
receivedfromtoccbccmessage_idin_reply_tocharacter_countword_countmessage_typeusersentimentsentiment_score
timestamp
2023-12-15 12:19:43+00:00NaT0[1][][]2<NA>336outgoingf1f97e90-2e1d-11ef-abd1-b0dcef010c43positive0.993223
2023-12-15 12:29:43+00:00NaT0[6, 1][][]1<NA>316outgoingf1f97e90-2e1d-11ef-abd1-b0dcef010c43negative0.980209
2023-12-15 12:29:43+00:00NaT0[6, 1][][]1<NA>285outgoingf1f97e90-2e1d-11ef-abd1-b0dcef010c43negative0.968588
2023-12-15 12:39:43+00:002023-12-15 12:19:43+00:006[0][4][4]32305incomingf1f97e90-2e1d-11ef-abd1-b0dcef010c43positive0.997529
2023-12-15 12:39:43+00:00Sat, 15 DeNot a timec 2023 12:19:43 00006[0][4][4]32517incomingf1f97e90-2e1d-11ef-abd1-b0dcef010c43neutral0.477333
\n", "" ], "text/plain": [ " received from \\\n", "timestamp \n", "2023-12-15 12:19:43+00:00 NaT 0 \n", "2023-12-15 12:29:43+00:00 NaT 0 \n", "2023-12-15 12:29:43+00:00 NaT 0 \n", "2023-12-15 12:39:43+00:00 2023-12-15 12:19:43+00:00 6 \n", "2023-12-15 12:39:43+00:00 Sat, 15 DeNot a timec 2023 12:19:43 0000 6 \n", "\n", " to cc bcc message_id in_reply_to \\\n", "timestamp \n", "2023-12-15 12:19:43+00:00 [1] [] [] 2 \n", "2023-12-15 12:29:43+00:00 [6, 1] [] [] 1 \n", "2023-12-15 12:29:43+00:00 [6, 1] [] [] 1 \n", "2023-12-15 12:39:43+00:00 [0] [4] [4] 3 2 \n", "2023-12-15 12:39:43+00:00 [0] [4] [4] 3 2 \n", "\n", " character_count word_count message_type \\\n", "timestamp \n", "2023-12-15 12:19:43+00:00 33 6 outgoing \n", "2023-12-15 12:29:43+00:00 31 6 outgoing \n", "2023-12-15 12:29:43+00:00 28 5 outgoing \n", "2023-12-15 12:39:43+00:00 30 5 incoming \n", "2023-12-15 12:39:43+00:00 51 7 incoming \n", "\n", " user sentiment \\\n", "timestamp \n", "2023-12-15 12:19:43+00:00 f1f97e90-2e1d-11ef-abd1-b0dcef010c43 positive \n", "2023-12-15 12:29:43+00:00 f1f97e90-2e1d-11ef-abd1-b0dcef010c43 negative \n", "2023-12-15 12:29:43+00:00 f1f97e90-2e1d-11ef-abd1-b0dcef010c43 negative \n", "2023-12-15 12:39:43+00:00 f1f97e90-2e1d-11ef-abd1-b0dcef010c43 positive \n", "2023-12-15 12:39:43+00:00 f1f97e90-2e1d-11ef-abd1-b0dcef010c43 neutral \n", "\n", " sentiment_score \n", "timestamp \n", "2023-12-15 12:19:43+00:00 0.993223 \n", "2023-12-15 12:29:43+00:00 0.980209 \n", "2023-12-15 12:29:43+00:00 0.968588 \n", "2023-12-15 12:39:43+00:00 0.997529 \n", "2023-12-15 12:39:43+00:00 0.477333 " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "path = os.path.join(config.GOOGLE_TAKEOUT_DIR, \"Takeout\", \"Mail\", \"All mail Including Spam and Trash.mbox\")\n", "niimpy.reading.google_takeout.email_activity(path, sentiment=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### General Notes on Google Takeout\n", "\n", "Each subject Downloads their Google TakeOut data as a separate zip file. The Zipfile package, which is included in the Python standard, is convenient for reading the data files contained in the zip file. For example, one could read the location data with the following code:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
latitudeE7longitudeE7accuracysourcedeviceTagtimestampactivitylocationMetadataplaceIdformFactor...deviceDesignationaltitudeverticalAccuracyplatformTypeosLevelserverTimestampdeviceTimestampbatteryChargingvelocityheading
0359974880-78922194325WIFI-5776802602016-08-12T19:29:43.821ZNaNNaNNaNNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
1359975588-78922503621WIFI-5776802602016-08-12T19:30:49.531Z[{'activity': [{'type': 'STILL', 'confidence':...NaNNaNNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
2359975588-78922503621WIFI-5776802602016-08-12T19:31:49.531Z[{'activity': [{'type': 'STILL', 'confidence':...[{'wifiScan': {'accessPoints': [{'mac': '12410...ChIJS_5Nmuz1jUYRGYf3QiiZco4PHONE...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
3360008703-7892334331500CELL-5776802602016-08-12T21:15:55.295Z[{'activity': [{'type': 'ON_FOOT', 'confidence...NaNNaNNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
4359972502-7892398948GPS-5776802602016-08-12T21:16:33ZNaNNaNNaNNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
\n", "

5 rows × 22 columns

\n", "
" ], "text/plain": [ " latitudeE7 longitudeE7 accuracy source deviceTag \\\n", "0 359974880 -789221943 25 WIFI -577680260 \n", "1 359975588 -789225036 21 WIFI -577680260 \n", "2 359975588 -789225036 21 WIFI -577680260 \n", "3 360008703 -789233433 1500 CELL -577680260 \n", "4 359972502 -789239894 8 GPS -577680260 \n", "\n", " timestamp \\\n", "0 2016-08-12T19:29:43.821Z \n", "1 2016-08-12T19:30:49.531Z \n", "2 2016-08-12T19:31:49.531Z \n", "3 2016-08-12T21:15:55.295Z \n", "4 2016-08-12T21:16:33Z \n", "\n", " activity \\\n", "0 NaN \n", "1 [{'activity': [{'type': 'STILL', 'confidence':... \n", "2 [{'activity': [{'type': 'STILL', 'confidence':... \n", "3 [{'activity': [{'type': 'ON_FOOT', 'confidence... \n", "4 NaN \n", "\n", " locationMetadata \\\n", "0 NaN \n", "1 NaN \n", "2 [{'wifiScan': {'accessPoints': [{'mac': '12410... \n", "3 NaN \n", "4 NaN \n", "\n", " placeId formFactor ... deviceDesignation altitude \\\n", "0 NaN NaN ... NaN NaN \n", "1 NaN NaN ... NaN NaN \n", "2 ChIJS_5Nmuz1jUYRGYf3QiiZco4 PHONE ... NaN NaN \n", "3 NaN NaN ... NaN NaN \n", "4 NaN NaN ... NaN NaN \n", "\n", " verticalAccuracy platformType osLevel serverTimestamp deviceTimestamp \\\n", "0 NaN NaN NaN NaN NaN \n", "1 NaN NaN NaN NaN NaN \n", "2 NaN NaN NaN NaN NaN \n", "3 NaN NaN NaN NaN NaN \n", "4 NaN NaN NaN NaN NaN \n", "\n", " batteryCharging velocity heading \n", "0 NaN NaN NaN \n", "1 NaN NaN NaN \n", "2 NaN NaN NaN \n", "3 NaN NaN NaN \n", "4 NaN NaN NaN \n", "\n", "[5 rows x 22 columns]" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from zipfile import ZipFile\n", "import json\n", "import pandas as pd\n", "\n", "zip_file = ZipFile(\"test.zip\")\n", "json_data = zip_file.read(\"Takeout/Location History (Timeline)/Records.json\")\n", "json_data = json.loads(json_data)\n", "data = pd.json_normalize(json_data[\"locations\"])\n", "data = pd.DataFrame(data)\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Location data is stored in the json format. Other types of data are stored in various formats and with different files structures. The user must find how each type of data they need is stored and how it can be read in Python." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## MHealth\n", "\n", "We have implemented readers for 3 data types formatted according to the [MHealth schema](https://www.openmhealth.org/documentation/#/schema-docs/schema-library). These are total sleep time, heart rate and geolocation. Other data types may be added as needed." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
descriptive_statisticdescriptive_statistic_denominatordatepart_of_daytotal_sleep_timestartend
timestamp
2016-02-06 04:35:00+00:00NaNNaNNaTNaN0 days 07:45:002016-02-06 04:35:00+00:002016-02-06 14:35:00+00:00
2016-02-05 15:00:00+00:00averagedNaTNaN0 days 07:15:002016-02-05 15:00:00+00:002016-06-06 15:00:00+00:00
2013-01-26 07:35:00+00:00NaNNaNNaTNaN0 days 03:00:002013-01-26 07:35:00+00:002013-02-05 07:35:00+00:00
2013-02-05 00:00:00+00:00NaNNaN2013-02-05 00:00:00+00:00evening0 days 03:00:00NaTNaT
\n", "
" ], "text/plain": [ " descriptive_statistic \\\n", "timestamp \n", "2016-02-06 04:35:00+00:00 NaN \n", "2016-02-05 15:00:00+00:00 average \n", "2013-01-26 07:35:00+00:00 NaN \n", "2013-02-05 00:00:00+00:00 NaN \n", "\n", " descriptive_statistic_denominator \\\n", "timestamp \n", "2016-02-06 04:35:00+00:00 NaN \n", "2016-02-05 15:00:00+00:00 d \n", "2013-01-26 07:35:00+00:00 NaN \n", "2013-02-05 00:00:00+00:00 NaN \n", "\n", " date part_of_day \\\n", "timestamp \n", "2016-02-06 04:35:00+00:00 NaT NaN \n", "2016-02-05 15:00:00+00:00 NaT NaN \n", "2013-01-26 07:35:00+00:00 NaT NaN \n", "2013-02-05 00:00:00+00:00 2013-02-05 00:00:00+00:00 evening \n", "\n", " total_sleep_time start \\\n", "timestamp \n", "2016-02-06 04:35:00+00:00 0 days 07:45:00 2016-02-06 04:35:00+00:00 \n", "2016-02-05 15:00:00+00:00 0 days 07:15:00 2016-02-05 15:00:00+00:00 \n", "2013-01-26 07:35:00+00:00 0 days 03:00:00 2013-01-26 07:35:00+00:00 \n", "2013-02-05 00:00:00+00:00 0 days 03:00:00 NaT \n", "\n", " end \n", "timestamp \n", "2016-02-06 04:35:00+00:00 2016-02-06 14:35:00+00:00 \n", "2016-02-05 15:00:00+00:00 2016-06-06 15:00:00+00:00 \n", "2013-01-26 07:35:00+00:00 2013-02-05 07:35:00+00:00 \n", "2013-02-05 00:00:00+00:00 NaT " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Reading total sleep time data:\n", "filename = config.MHEALTH_TOTAL_SLEEP_TIME_PATH\n", "niimpy.reading.mhealth.total_sleep_time_from_file(filename)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
temporal_relationship_to_sleepheart_rateeffective_time_frame.date_timedescriptive_statisticstartend
timestamp
NaTon waking702023-11-20T07:25:00-08:00NaNNaTNaT
2023-12-20 09:50:00+00:00on waking65NaNNaN2023-12-20 09:50:00+00:002023-12-20 10:00:00+00:00
2023-12-20 03:50:00+00:00during sleep35NaNaverage2023-12-20 03:50:00+00:002023-12-20 04:00:00+00:00
\n", "
" ], "text/plain": [ " temporal_relationship_to_sleep heart_rate \\\n", "timestamp \n", "NaT on waking 70 \n", "2023-12-20 09:50:00+00:00 on waking 65 \n", "2023-12-20 03:50:00+00:00 during sleep 35 \n", "\n", " effective_time_frame.date_time \\\n", "timestamp \n", "NaT 2023-11-20T07:25:00-08:00 \n", "2023-12-20 09:50:00+00:00 NaN \n", "2023-12-20 03:50:00+00:00 NaN \n", "\n", " descriptive_statistic start \\\n", "timestamp \n", "NaT NaN NaT \n", "2023-12-20 09:50:00+00:00 NaN 2023-12-20 09:50:00+00:00 \n", "2023-12-20 03:50:00+00:00 average 2023-12-20 03:50:00+00:00 \n", "\n", " end \n", "timestamp \n", "NaT NaT \n", "2023-12-20 09:50:00+00:00 2023-12-20 10:00:00+00:00 \n", "2023-12-20 03:50:00+00:00 2023-12-20 04:00:00+00:00 " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Reading heart rate data:\n", "filename = config.MHEALTH_HEART_RATE_PATH\n", "niimpy.reading.mhealth.heart_rate_from_file(filename)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
positioning_systemlatitudelatitude.unitlongitudelongitude.uniteffective_time_frame.time_interval.start_date_timeeffective_time_frame.time_interval.end_date_timeelevation.valueelevation.unit
0GPS60.1867deg24.8283deg2016-02-05T20:35:00-08:002016-02-06T06:35:00-08:00NaNNaN
1GPS60.1867deg24.8283deg2016-02-05T20:35:00-08:002016-02-06T06:35:00-08:0020.4m
\n", "
" ], "text/plain": [ " positioning_system latitude latitude.unit longitude longitude.unit \\\n", "0 GPS 60.1867 deg 24.8283 deg \n", "1 GPS 60.1867 deg 24.8283 deg \n", "\n", " effective_time_frame.time_interval.start_date_time \\\n", "0 2016-02-05T20:35:00-08:00 \n", "1 2016-02-05T20:35:00-08:00 \n", "\n", " effective_time_frame.time_interval.end_date_time elevation.value \\\n", "0 2016-02-06T06:35:00-08:00 NaN \n", "1 2016-02-06T06:35:00-08:00 20.4 \n", "\n", " elevation.unit \n", "0 NaN \n", "1 m " ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Reading geolocation data:\n", "filename = config.MHEALTH_GEOLOCATION_PATH\n", "niimpy.reading.mhealth.geolocation_from_file(filename)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Other formats\n", "\n", "You can add readers for any types of formats which you can convert into a Pandas dataframe (so basically anything). For examples of readers, see `niimpy/reading/read.py`. Apply the function `niimpy.preprocessing.util.df_normalize` in order to apply some standardizations to get the standard Niimpy format." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.6" } }, "nbformat": 4, "nbformat_minor": 4 }