{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "25315fc6",
"metadata": {},
"source": [
"# Screen On/Off Data"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "5daa4ada",
"metadata": {},
"source": [
"## Introduction\n",
"\n",
"Screen data refers to the information about the status of the screen as reported by Android. These data can reveal important information about people's circadian rhythm, social patterns, and activity. Screen data is an event data, this means that it cannot be sampled at a regular frequency. We just have information about the events that occured.\n",
"\n",
"A dataframe with screen data should contain the following columns (column names can be different, but in that case they must be provided as parameters):\n",
"- `user`: Subject ID\n",
"- `device`: Device ID\n",
"- `screen_status`: An integer indicating screen new status. https://awareframework.com/battery/\n",
" - Status `0`: off\n",
" - Status `1`: on\n",
" - Status `2`: locked\n",
" - Status `3`: unlocked\n",
"\n",
"\n",
"Some factors may interfere with the correct detection of all screen events (e.g. when the phone's battery is depleated). Therefore, to correctly process screen data, we need to take into account other information like the battery status of the phone. This may complicate the preprocessing. To address this, `niimpy` includes a set of functions to clean, downsample, and extract features from screen data while taking into account factors like the battery level. The functions allow us to extract the following features:\n",
"\n",
"- `screen_off`: reports when the screen has been turned off\n",
"- `screen_count`: number of times the screen has turned on, off, or has been in use\n",
"- `screen_duration`: duration in seconds of the screen on, off, and in use statuses\n",
"- `screen_duration_min`: minimum duration in seconds of the screen on, off, and in use statuses\n",
"- `screen_duration_max`: maximum duration in seconds of the screen on, off, and in use statuses\n",
"- `screen_duration_median`: median duration in seconds of the screen on, off, and in use statuses\n",
"- `screen_duration_mean`: mean duration in seconds of the screen on, off, and in use statuses\n",
"- `screen_duration_std`: standard deviation of the duration in seconds of the screen on, off, and in use statuses\n",
"- `screen_first_unlock`: reports the first time when the phone was unlocked every day\n",
"- `extract_features_screen`: wrapper-like function to extract several features at the same time\n",
"\n",
"In addition, the screen module has three internal functions that help classify the events and calculate their status duration. \n",
"\n",
"In the following, we will analyze screen data provided by `niimpy` as an example to illustrate the use of screen data."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "483cbc81",
"metadata": {},
"source": [
"## 2. Read data\n",
"\n",
"Let's start by reading the example data provided in `niimpy`. These data have already been shaped in a format that meets the requirements of the data schema. Let's start by importing the needed modules. Firstly we will import the `niimpy` package and then we will import the module we will use (screen) and give it a short name for use convenience. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "67472b6c",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/u/24/rantahj1/unix/miniconda3/envs/niimpy/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n"
]
}
],
"source": [
"import niimpy\n",
"from niimpy import config\n",
"import niimpy.preprocessing.screen as s\n",
"import pandas as pd\n",
"import warnings\n",
"warnings.filterwarnings(\"ignore\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "39457805",
"metadata": {},
"source": [
"Now let's read the example data provided in `niimpy`. The example data is in `csv` format, so we need to use the `read_csv` function. When reading the data, we can specify the timezone where the data was collected. This will help us handle daylight saving times easier. We can specify the timezone with the argument **tz**. The output is a dataframe. We can also check the number of rows and columns in the dataframe."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "d997fcf2",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(343, 5)"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = niimpy.read_csv(config.MULTIUSER_AWARE_SCREEN_PATH, tz='Europe/Helsinki')\n",
"data.shape"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "686f5d7e",
"metadata": {},
"source": [
"The data was succesfully read. We can see that there are 277 datapoints with 5 columns in the dataset. However, we do not know yet what the data really looks like, so let's have a quick look: "
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "3867b401",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" device | \n",
" time | \n",
" screen_status | \n",
" datetime | \n",
"
\n",
" \n",
" \n",
" \n",
" 2020-01-09 02:06:41.573999882+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.578528e+09 | \n",
" 0 | \n",
" 2020-01-09 02:06:41.573999882+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:09:29.151999950+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.578529e+09 | \n",
" 1 | \n",
" 2020-01-09 02:09:29.151999950+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:09:32.790999889+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.578529e+09 | \n",
" 3 | \n",
" 2020-01-09 02:09:32.790999889+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:11:41.996000051+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.578529e+09 | \n",
" 0 | \n",
" 2020-01-09 02:11:41.996000051+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:16:19.010999918+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.578529e+09 | \n",
" 1 | \n",
" 2020-01-09 02:16:19.010999918+02:00 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user device time \\\n",
"2020-01-09 02:06:41.573999882+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.578528e+09 \n",
"2020-01-09 02:09:29.151999950+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.578529e+09 \n",
"2020-01-09 02:09:32.790999889+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.578529e+09 \n",
"2020-01-09 02:11:41.996000051+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.578529e+09 \n",
"2020-01-09 02:16:19.010999918+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.578529e+09 \n",
"\n",
" screen_status \\\n",
"2020-01-09 02:06:41.573999882+02:00 0 \n",
"2020-01-09 02:09:29.151999950+02:00 1 \n",
"2020-01-09 02:09:32.790999889+02:00 3 \n",
"2020-01-09 02:11:41.996000051+02:00 0 \n",
"2020-01-09 02:16:19.010999918+02:00 1 \n",
"\n",
" datetime \n",
"2020-01-09 02:06:41.573999882+02:00 2020-01-09 02:06:41.573999882+02:00 \n",
"2020-01-09 02:09:29.151999950+02:00 2020-01-09 02:09:29.151999950+02:00 \n",
"2020-01-09 02:09:32.790999889+02:00 2020-01-09 02:09:32.790999889+02:00 \n",
"2020-01-09 02:11:41.996000051+02:00 2020-01-09 02:11:41.996000051+02:00 \n",
"2020-01-09 02:16:19.010999918+02:00 2020-01-09 02:16:19.010999918+02:00 "
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data.head()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "c659d5a7",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" device | \n",
" time | \n",
" screen_status | \n",
" datetime | \n",
"
\n",
" \n",
" \n",
" \n",
" 2019-08-07 17:42:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 1.565189e+09 | \n",
" 2 | \n",
" 2019-08-07 17:42:41.009999990+03:00 | \n",
"
\n",
" \n",
" 2019-08-07 18:32:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 1.565192e+09 | \n",
" 1 | \n",
" 2019-08-07 18:32:41.009999990+03:00 | \n",
"
\n",
" \n",
" 2019-08-07 19:22:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 1.565195e+09 | \n",
" 0 | \n",
" 2019-08-07 19:22:41.009999990+03:00 | \n",
"
\n",
" \n",
" 2019-08-07 20:12:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 1.565198e+09 | \n",
" 1 | \n",
" 2019-08-07 20:12:41.009999990+03:00 | \n",
"
\n",
" \n",
" 2019-08-07 21:02:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 1.565201e+09 | \n",
" 2 | \n",
" 2019-08-07 21:02:41.009999990+03:00 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user device \\\n",
"2019-08-07 17:42:41.009999990+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-07 18:32:41.009999990+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-07 19:22:41.009999990+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-07 20:12:41.009999990+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-07 21:02:41.009999990+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"\n",
" time screen_status \\\n",
"2019-08-07 17:42:41.009999990+03:00 1.565189e+09 2 \n",
"2019-08-07 18:32:41.009999990+03:00 1.565192e+09 1 \n",
"2019-08-07 19:22:41.009999990+03:00 1.565195e+09 0 \n",
"2019-08-07 20:12:41.009999990+03:00 1.565198e+09 1 \n",
"2019-08-07 21:02:41.009999990+03:00 1.565201e+09 2 \n",
"\n",
" datetime \n",
"2019-08-07 17:42:41.009999990+03:00 2019-08-07 17:42:41.009999990+03:00 \n",
"2019-08-07 18:32:41.009999990+03:00 2019-08-07 18:32:41.009999990+03:00 \n",
"2019-08-07 19:22:41.009999990+03:00 2019-08-07 19:22:41.009999990+03:00 \n",
"2019-08-07 20:12:41.009999990+03:00 2019-08-07 20:12:41.009999990+03:00 \n",
"2019-08-07 21:02:41.009999990+03:00 2019-08-07 21:02:41.009999990+03:00 "
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data.tail()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "55f3f442",
"metadata": {},
"source": [
"By exploring the head and tail of the dataframe we can form an idea of its entirety. From the data, we can see that:\n",
"\n",
"- rows are observations, indexed by timestamps, i.e. each row represents a screen event at a given time and date\n",
"- columns are characteristics for each observation, for example, the user whose data we are analyzing\n",
"- there are at least two different users in the dataframe\n",
"- the main column is `screen_status`. This screen status is coded in numbers as: 0=off, 1=on, 2=locked, 3=unlocked.\n",
"\n",
"## * TIP! Data format requirements (or what should our data look like)\n",
"\n",
"Data can take other shapes and formats. However, the `niimpy` data scheme requires it to be in a certain shape. This means the dataframe needs to have at least the following characteristics:\n",
"1. One row per screen status. Each row should store information about one screen status only\n",
"2. Each row's index should be a timestamp\n",
"3. There should be at least three columns: \n",
" - index: date and time when the event happened (timestamp)\n",
" - user: stores the user name whose data is analyzed. Each user should have a unique name or hash (i.e. one hash for each unique user)\n",
" - screen_status: stores the screen status (0,1,2, or 3) as defined by Android.\n",
"4. Columns additional to those listed in item 3 are allowed\n",
"5. The names of the columns do not need to be exactly \"screen_status\" as we can pass our own names in an argument (to be explained later).\n",
"\n",
"Below is an example of a dataframe that complies with these minimum requirements"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "2dc89a1e",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" screen_status | \n",
"
\n",
" \n",
" \n",
" \n",
" 2020-01-09 02:06:41.573999882+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 0 | \n",
"
\n",
" \n",
" 2020-01-09 02:09:29.151999950+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 1 | \n",
"
\n",
" \n",
" 2020-01-09 02:09:32.790999889+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 3 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user screen_status\n",
"2020-01-09 02:06:41.573999882+02:00 jd9INuQ5BBlW 0\n",
"2020-01-09 02:09:29.151999950+02:00 jd9INuQ5BBlW 1\n",
"2020-01-09 02:09:32.790999889+02:00 jd9INuQ5BBlW 3"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example_dataschema = data[['user','screen_status']]\n",
"example_dataschema.head(3)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "fc94c6d7",
"metadata": {},
"source": [
"#### A few words on missing data\n",
"Missing data for screen is difficult to detect. Firstly, this sensor is triggered by events and not sampled at a fixed frequency. Secondly, different phones, OS, and settings change how the screen is turned on/off; for example, one phone may go from OFF to ON to UNLOCKED, while another phone may go from OFF to UNLOCKED directly. Thirdly, events not related to the screen may affect its behavior, e.g. battery running out. Neverthless, there are some events transitions that are impossible to have, like a status to itself (e.g. two consecutive 0s). These *imposible* statuses helps us determine the missing data. \n",
"\n",
"#### A few words on the classification of the events\n",
"We can know the status of the screen at a certain timepoint. However, we need a bit more to know the duration and the meaning of it. Consequently, we need to look at the numbers of two consecutive events and classify the transitions (going from one state to another consecutively) as:\n",
"- from 3 to 0,1,2: the phone was in use \n",
"- from 1 to 0,1,3: the phone was on\n",
"- from 0 to 1,2,3: the phone was off\n",
"\n",
"Other transitions are irrelevant. \n",
"\n",
"#### A few words on the role of the battery\n",
"As mentioned before, battery statuses can affect the screen behavior. In particular, when the battery is depleated and the phone is shut down automatically, the screen sensor does not cast any events, so even when the screen is technically OFF because the phone does not have any battery left, we will not see that 0 in the screen status column. Thus, it is important to take into account the battery information when analyzing screen data. `niimpy`'s screen module is adapted to take into account the battery data. \n",
"Since we do have some battery data, we will load it."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "e5594353",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" device | \n",
" time | \n",
" battery_level | \n",
" battery_status | \n",
" battery_health | \n",
" battery_adaptor | \n",
" datetime | \n",
"
\n",
" \n",
" \n",
" \n",
" 2020-01-09 02:20:02.924999952+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 3p83yASkOb_B | \n",
" 1.578529e+09 | \n",
" 74 | \n",
" 3 | \n",
" 2 | \n",
" 0 | \n",
" 2020-01-09 02:20:02.924999952+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:21:30.405999899+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 3p83yASkOb_B | \n",
" 1.578529e+09 | \n",
" 73 | \n",
" 3 | \n",
" 2 | \n",
" 0 | \n",
" 2020-01-09 02:21:30.405999899+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:24:12.805999994+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 3p83yASkOb_B | \n",
" 1.578529e+09 | \n",
" 72 | \n",
" 3 | \n",
" 2 | \n",
" 0 | \n",
" 2020-01-09 02:24:12.805999994+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:35:38.561000109+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 3p83yASkOb_B | \n",
" 1.578530e+09 | \n",
" 72 | \n",
" 2 | \n",
" 2 | \n",
" 0 | \n",
" 2020-01-09 02:35:38.561000109+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:35:38.953000069+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 3p83yASkOb_B | \n",
" 1.578530e+09 | \n",
" 72 | \n",
" 2 | \n",
" 2 | \n",
" 2 | \n",
" 2020-01-09 02:35:38.953000069+02:00 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user device time \\\n",
"2020-01-09 02:20:02.924999952+02:00 jd9INuQ5BBlW 3p83yASkOb_B 1.578529e+09 \n",
"2020-01-09 02:21:30.405999899+02:00 jd9INuQ5BBlW 3p83yASkOb_B 1.578529e+09 \n",
"2020-01-09 02:24:12.805999994+02:00 jd9INuQ5BBlW 3p83yASkOb_B 1.578529e+09 \n",
"2020-01-09 02:35:38.561000109+02:00 jd9INuQ5BBlW 3p83yASkOb_B 1.578530e+09 \n",
"2020-01-09 02:35:38.953000069+02:00 jd9INuQ5BBlW 3p83yASkOb_B 1.578530e+09 \n",
"\n",
" battery_level battery_status \\\n",
"2020-01-09 02:20:02.924999952+02:00 74 3 \n",
"2020-01-09 02:21:30.405999899+02:00 73 3 \n",
"2020-01-09 02:24:12.805999994+02:00 72 3 \n",
"2020-01-09 02:35:38.561000109+02:00 72 2 \n",
"2020-01-09 02:35:38.953000069+02:00 72 2 \n",
"\n",
" battery_health battery_adaptor \\\n",
"2020-01-09 02:20:02.924999952+02:00 2 0 \n",
"2020-01-09 02:21:30.405999899+02:00 2 0 \n",
"2020-01-09 02:24:12.805999994+02:00 2 0 \n",
"2020-01-09 02:35:38.561000109+02:00 2 0 \n",
"2020-01-09 02:35:38.953000069+02:00 2 2 \n",
"\n",
" datetime \n",
"2020-01-09 02:20:02.924999952+02:00 2020-01-09 02:20:02.924999952+02:00 \n",
"2020-01-09 02:21:30.405999899+02:00 2020-01-09 02:21:30.405999899+02:00 \n",
"2020-01-09 02:24:12.805999994+02:00 2020-01-09 02:24:12.805999994+02:00 \n",
"2020-01-09 02:35:38.561000109+02:00 2020-01-09 02:35:38.561000109+02:00 \n",
"2020-01-09 02:35:38.953000069+02:00 2020-01-09 02:35:38.953000069+02:00 "
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"bat_data = niimpy.read_csv(config.MULTIUSER_AWARE_BATTERY_PATH, tz='Europe/Helsinki')\n",
"bat_data.head()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "314937cb",
"metadata": {},
"source": [
"In this case, we are interested in the battery_status information. This is standard information provided by Android. However, if the dataframe has this information in a column with a different name, we can use the argument `battery_column_name` similarly to the use of `screen_column_name` (more info about this topic below)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b10cde9b",
"metadata": {},
"source": [
"## 4. Extracting features\n",
"There are two ways to extract features. We could use each function separately or we could use `niimpy`'s ready-made wrapper. Both ways will require us to specify arguments to pass to the functions/wrapper in order to customize the way the functions work. These arguments are specified in dictionaries. Let's first understand how to extract features using stand-alone functions.\n",
"\n",
"### 4.1 Extract features using stand-alone functions\n",
"We can use `niimpy`'s functions to compute communication features. Each function will require two inputs:\n",
"- (mandatory) dataframe that must comply with the minimum requirements (see '* TIP! Data requirements above)\n",
"- (optional) arguments for stand-alone functions\n",
"\n",
"#### 4.1.1 The argument dictionary for stand-alone functions (or how we specify the way a function works)\n",
"We can input two types of arguments to customize the way a stand-alone function works:\n",
"- the name of the columns to be preprocessed: Since the dataframe may have different columns, we need to specify which column has the data we would like to be preprocessed. To do so, we can simply pass the name of the column to the argument `screen_column_name`. \n",
"\n",
"- the way we resample: resampling options are specified in `niimpy` as a dictionary. `niimpy`'s resampling and aggregating relies on `pandas.DataFrame.resample`, so mastering the use of this pandas function will help us greatly in `niimpy`'s preprocessing. Please familiarize yourself with the pandas resample function before continuing. \n",
" Briefly, to use the `pandas.DataFrame.resample` function, we need a rule. This rule states the intervals we would like to use to resample our data (e.g., 15-seconds, 30-minutes, 1-hour). Neverthless, we can input more details into the function to specify the exact sampling we would like. For example, we could use the *close* argument if we would like to specify which side of the interval is closed, or we could use the *offset* argument if we would like to start our binning with an offset, etc. There are plenty of options to use this command, so we strongly recommend having `pandas.DataFrame.resample` documentation at hand. All features for the `pandas.DataFrame.resample` will be specified in a dictionary where keys are the arguments' names for the `pandas.DataFrame.resample`, and the dictionary's values are the values for each of these selected arguments. This dictionary will be passed as a value to the key `resample_args` in `niimpy`.\n",
"\n",
"Let's see some basic examples of these dictionaries:"
]
},
{
"cell_type": "markdown",
"id": "c4af6a7b",
"metadata": {},
"source": [
"```python\n",
"s.screen_count(data, screen_column_name = \"screen_status\", resample_args = {\"rule\":\"1D\"})\n",
"s.screen_count(data, screen_column_name = \"random_name\", resample_args = {\"rule\":\"30T\"})\n",
"s.screen_count(data, screen_column_name = \"other_name\", resample_args = {\"rule\":\"45T\",\"origin\":\"end\"})\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "c70f0157",
"metadata": {},
"source": [
"Here, we have three basic feature dictionaries. \n",
"\n",
"- The first example will analyze the data stored in the column `screen_status` in our dataframe. The data will be binned in one day periods\n",
"- The second example will analyze the data stored in the column `random_name` in our dataframe. The data will be aggregated in 30-minutes bins\n",
"- The third example will analyze the data stored in the column `other_name` in our dataframe. The data will be binned in 45-minutes bins, but the binning will start from the last timestamp in the dataframe. \n",
"\n",
"**Default values:** if no arguments are passed, `niimpy`'s default values are \"screen_status\" for the screen_column_name, and 30-min aggregation bins. "
]
},
{
"cell_type": "markdown",
"id": "7e336fe0",
"metadata": {},
"source": [
"#### 4.1.2 Using the functions\n",
"Now that we understand how the functions are customized, it is time we compute our first communication feature. Suppose that we are interested in extracting the total duration of outgoing calls every 20 minutes. We will need `niimpy`'s `screen_count` function."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "f1ccfcc9",
"metadata": {},
"outputs": [],
"source": [
"my_screen_count = s.screen_count(data, bat_data, screen_column_name = \"screen_status\", resample_args = {\"rule\":\"20T\"})"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b97a9da2",
"metadata": {},
"source": [
"Let's look at some values for one of the subjects."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "353d5acc",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" screen_on_count | \n",
" screen_off_count | \n",
" screen_use_count | \n",
" device | \n",
"
\n",
" \n",
" \n",
" \n",
" 2020-01-09 02:00:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 2 | \n",
" 2 | \n",
" 2 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" 2020-01-09 02:20:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 3 | \n",
" 4 | \n",
" 2 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" 2020-01-09 02:40:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 2 | \n",
" 2 | \n",
" 1 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" 2020-01-09 03:00:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" 2020-01-09 03:20:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
"
\n",
" \n",
" 2020-01-09 21:40:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 1 | \n",
" 1 | \n",
" 0 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" 2020-01-09 22:00:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 1 | \n",
" 1 | \n",
" 0 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" 2020-01-09 22:20:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" 2020-01-09 22:40:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" 2020-01-09 23:00:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 4 | \n",
" 3 | \n",
" 0 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
"
\n",
"
64 rows × 5 columns
\n",
"
"
],
"text/plain": [
" user screen_on_count screen_off_count \\\n",
"2020-01-09 02:00:00+02:00 jd9INuQ5BBlW 2 2 \n",
"2020-01-09 02:20:00+02:00 jd9INuQ5BBlW 3 4 \n",
"2020-01-09 02:40:00+02:00 jd9INuQ5BBlW 2 2 \n",
"2020-01-09 03:00:00+02:00 jd9INuQ5BBlW 0 0 \n",
"2020-01-09 03:20:00+02:00 jd9INuQ5BBlW 0 0 \n",
"... ... ... ... \n",
"2020-01-09 21:40:00+02:00 jd9INuQ5BBlW 1 1 \n",
"2020-01-09 22:00:00+02:00 jd9INuQ5BBlW 1 1 \n",
"2020-01-09 22:20:00+02:00 jd9INuQ5BBlW 0 0 \n",
"2020-01-09 22:40:00+02:00 jd9INuQ5BBlW 0 0 \n",
"2020-01-09 23:00:00+02:00 jd9INuQ5BBlW 4 3 \n",
"\n",
" screen_use_count device \n",
"2020-01-09 02:00:00+02:00 2 OWd1Uau8POix \n",
"2020-01-09 02:20:00+02:00 2 OWd1Uau8POix \n",
"2020-01-09 02:40:00+02:00 1 OWd1Uau8POix \n",
"2020-01-09 03:00:00+02:00 0 OWd1Uau8POix \n",
"2020-01-09 03:20:00+02:00 0 OWd1Uau8POix \n",
"... ... ... \n",
"2020-01-09 21:40:00+02:00 0 OWd1Uau8POix \n",
"2020-01-09 22:00:00+02:00 0 OWd1Uau8POix \n",
"2020-01-09 22:20:00+02:00 0 OWd1Uau8POix \n",
"2020-01-09 22:40:00+02:00 0 OWd1Uau8POix \n",
"2020-01-09 23:00:00+02:00 0 OWd1Uau8POix \n",
"\n",
"[64 rows x 5 columns]"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"my_screen_count[my_screen_count[\"user\"] == \"jd9INuQ5BBlW\"]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "9b0f177b",
"metadata": {},
"source": [
"Let's remember how the original data looked like for this subject"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "a906348d",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" device | \n",
" time | \n",
" screen_status | \n",
" datetime | \n",
"
\n",
" \n",
" \n",
" \n",
" 2020-01-09 02:06:41.573999882+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.578528e+09 | \n",
" 0 | \n",
" 2020-01-09 02:06:41.573999882+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:09:29.151999950+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.578529e+09 | \n",
" 1 | \n",
" 2020-01-09 02:09:29.151999950+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:09:32.790999889+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.578529e+09 | \n",
" 3 | \n",
" 2020-01-09 02:09:32.790999889+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:11:41.996000051+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.578529e+09 | \n",
" 0 | \n",
" 2020-01-09 02:11:41.996000051+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:16:19.010999918+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.578529e+09 | \n",
" 1 | \n",
" 2020-01-09 02:16:19.010999918+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:16:29.648999929+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.578529e+09 | \n",
" 0 | \n",
" 2020-01-09 02:16:29.648999929+02:00 | \n",
"
\n",
" \n",
" 2020-01-09 02:16:29.657999992+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.578529e+09 | \n",
" 2 | \n",
" 2020-01-09 02:16:29.657999992+02:00 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user device time \\\n",
"2020-01-09 02:06:41.573999882+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.578528e+09 \n",
"2020-01-09 02:09:29.151999950+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.578529e+09 \n",
"2020-01-09 02:09:32.790999889+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.578529e+09 \n",
"2020-01-09 02:11:41.996000051+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.578529e+09 \n",
"2020-01-09 02:16:19.010999918+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.578529e+09 \n",
"2020-01-09 02:16:29.648999929+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.578529e+09 \n",
"2020-01-09 02:16:29.657999992+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.578529e+09 \n",
"\n",
" screen_status \\\n",
"2020-01-09 02:06:41.573999882+02:00 0 \n",
"2020-01-09 02:09:29.151999950+02:00 1 \n",
"2020-01-09 02:09:32.790999889+02:00 3 \n",
"2020-01-09 02:11:41.996000051+02:00 0 \n",
"2020-01-09 02:16:19.010999918+02:00 1 \n",
"2020-01-09 02:16:29.648999929+02:00 0 \n",
"2020-01-09 02:16:29.657999992+02:00 2 \n",
"\n",
" datetime \n",
"2020-01-09 02:06:41.573999882+02:00 2020-01-09 02:06:41.573999882+02:00 \n",
"2020-01-09 02:09:29.151999950+02:00 2020-01-09 02:09:29.151999950+02:00 \n",
"2020-01-09 02:09:32.790999889+02:00 2020-01-09 02:09:32.790999889+02:00 \n",
"2020-01-09 02:11:41.996000051+02:00 2020-01-09 02:11:41.996000051+02:00 \n",
"2020-01-09 02:16:19.010999918+02:00 2020-01-09 02:16:19.010999918+02:00 \n",
"2020-01-09 02:16:29.648999929+02:00 2020-01-09 02:16:29.648999929+02:00 \n",
"2020-01-09 02:16:29.657999992+02:00 2020-01-09 02:16:29.657999992+02:00 "
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data[data[\"user\"]==\"jd9INuQ5BBlW\"].head(7)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "583aa994",
"metadata": {},
"source": [
"We see that the bins are indeed 20-minutes bins, however, they are adjusted to fixed, predetermined intervals, i.e. the bin does not start on the time of the first datapoint. Instead, `pandas` starts the binning at 00:00:00 of everyday and counts 20-minutes intervals from there. \n",
"\n",
"If we want the binning to start from the first datapoint in our dataset, we need the origin parameter and a for loop."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "670d20f8",
"metadata": {},
"outputs": [],
"source": [
"users = list(data['user'].unique())\n",
"results = []\n",
"for user in users:\n",
" start_time = data[data[\"user\"]==user].index.min()\n",
" results.append(s.screen_count(\n",
" data[data[\"user\"]==user],\n",
" bat_data[bat_data[\"user\"]==user],\n",
" screen_column_name = \"screen_status\",\n",
" resample_args = {\"rule\":\"20T\",\"origin\":start_time}\n",
" ))\n",
"my_screen_count = pd.concat(results)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "0b9030fa",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" screen_on_count | \n",
" screen_off_count | \n",
" screen_use_count | \n",
" device | \n",
"
\n",
" \n",
" \n",
" \n",
" 2020-01-09 02:06:41.573999882+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 4 | \n",
" 3 | \n",
" 3 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" 2020-01-09 02:26:41.573999882+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 2 | \n",
" 3 | \n",
" 1 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" 2020-01-09 02:46:41.573999882+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 2 | \n",
" 2 | \n",
" 1 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" 2020-01-09 03:06:41.573999882+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" 2020-01-09 03:26:41.573999882+02:00 | \n",
" jd9INuQ5BBlW | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" OWd1Uau8POix | \n",
"
\n",
" \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
" ... | \n",
"
\n",
" \n",
" 2019-08-07 17:52:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
" 2019-08-07 18:12:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
" 2019-08-07 18:32:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" 1 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
" 2019-08-07 18:52:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
" 2019-08-07 19:12:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" 0 | \n",
" 1 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
"
\n",
"
2689 rows × 5 columns
\n",
"
"
],
"text/plain": [
" user screen_on_count \\\n",
"2020-01-09 02:06:41.573999882+02:00 jd9INuQ5BBlW 4 \n",
"2020-01-09 02:26:41.573999882+02:00 jd9INuQ5BBlW 2 \n",
"2020-01-09 02:46:41.573999882+02:00 jd9INuQ5BBlW 2 \n",
"2020-01-09 03:06:41.573999882+02:00 jd9INuQ5BBlW 0 \n",
"2020-01-09 03:26:41.573999882+02:00 jd9INuQ5BBlW 0 \n",
"... ... ... \n",
"2019-08-07 17:52:41.009999990+03:00 dvWdLQesv21a 0 \n",
"2019-08-07 18:12:41.009999990+03:00 dvWdLQesv21a 0 \n",
"2019-08-07 18:32:41.009999990+03:00 dvWdLQesv21a 1 \n",
"2019-08-07 18:52:41.009999990+03:00 dvWdLQesv21a 0 \n",
"2019-08-07 19:12:41.009999990+03:00 dvWdLQesv21a 0 \n",
"\n",
" screen_off_count screen_use_count \\\n",
"2020-01-09 02:06:41.573999882+02:00 3 3 \n",
"2020-01-09 02:26:41.573999882+02:00 3 1 \n",
"2020-01-09 02:46:41.573999882+02:00 2 1 \n",
"2020-01-09 03:06:41.573999882+02:00 0 0 \n",
"2020-01-09 03:26:41.573999882+02:00 0 0 \n",
"... ... ... \n",
"2019-08-07 17:52:41.009999990+03:00 0 0 \n",
"2019-08-07 18:12:41.009999990+03:00 0 0 \n",
"2019-08-07 18:32:41.009999990+03:00 0 0 \n",
"2019-08-07 18:52:41.009999990+03:00 0 0 \n",
"2019-08-07 19:12:41.009999990+03:00 1 0 \n",
"\n",
" device \n",
"2020-01-09 02:06:41.573999882+02:00 OWd1Uau8POix \n",
"2020-01-09 02:26:41.573999882+02:00 OWd1Uau8POix \n",
"2020-01-09 02:46:41.573999882+02:00 OWd1Uau8POix \n",
"2020-01-09 03:06:41.573999882+02:00 OWd1Uau8POix \n",
"2020-01-09 03:26:41.573999882+02:00 OWd1Uau8POix \n",
"... ... \n",
"2019-08-07 17:52:41.009999990+03:00 i8jmoIuoe12Mo \n",
"2019-08-07 18:12:41.009999990+03:00 i8jmoIuoe12Mo \n",
"2019-08-07 18:32:41.009999990+03:00 i8jmoIuoe12Mo \n",
"2019-08-07 18:52:41.009999990+03:00 i8jmoIuoe12Mo \n",
"2019-08-07 19:12:41.009999990+03:00 i8jmoIuoe12Mo \n",
"\n",
"[2689 rows x 5 columns]"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"my_screen_count"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "fa3a2f9a",
"metadata": {},
"source": [
"The functions can also be called in absence of a config dictionary. In this case, the binning will be automatically set to 30-minutes."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "39979c25",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" screen_on_count | \n",
" screen_off_count | \n",
" screen_use_count | \n",
" device | \n",
"
\n",
" \n",
" \n",
" \n",
" 2019-08-05 15:30:00+03:00 | \n",
" dvWdLQesv21a | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
" 2019-08-05 16:00:00+03:00 | \n",
" dvWdLQesv21a | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
" 2019-08-05 16:30:00+03:00 | \n",
" dvWdLQesv21a | \n",
" 1 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
" 2019-08-05 17:00:00+03:00 | \n",
" dvWdLQesv21a | \n",
" 0 | \n",
" 1 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
" 2019-08-05 17:30:00+03:00 | \n",
" dvWdLQesv21a | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user screen_on_count screen_off_count \\\n",
"2019-08-05 15:30:00+03:00 dvWdLQesv21a 0 0 \n",
"2019-08-05 16:00:00+03:00 dvWdLQesv21a 0 0 \n",
"2019-08-05 16:30:00+03:00 dvWdLQesv21a 1 0 \n",
"2019-08-05 17:00:00+03:00 dvWdLQesv21a 0 1 \n",
"2019-08-05 17:30:00+03:00 dvWdLQesv21a 0 0 \n",
"\n",
" screen_use_count device \n",
"2019-08-05 15:30:00+03:00 0 i8jmoIuoe12Mo \n",
"2019-08-05 16:00:00+03:00 0 i8jmoIuoe12Mo \n",
"2019-08-05 16:30:00+03:00 0 i8jmoIuoe12Mo \n",
"2019-08-05 17:00:00+03:00 0 i8jmoIuoe12Mo \n",
"2019-08-05 17:30:00+03:00 0 i8jmoIuoe12Mo "
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"my_screen_count = s.screen_count(data, bat_data)\n",
"my_screen_count.head()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "5a60bcbc",
"metadata": {},
"source": [
"In case we do not have battery data, the functions can also be called without it. In this case, simply input an empty dataframe in the second position of the function. For example,"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "3235f028",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" screen_on_count | \n",
" screen_off_count | \n",
" screen_use_count | \n",
" device | \n",
"
\n",
" \n",
" \n",
" \n",
" 2019-08-05 15:32:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
" 2019-08-05 15:52:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
" 2019-08-05 16:12:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
" 2019-08-05 16:32:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" 1 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
" 2019-08-05 16:52:41.009999990+03:00 | \n",
" dvWdLQesv21a | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" i8jmoIuoe12Mo | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user screen_on_count \\\n",
"2019-08-05 15:32:41.009999990+03:00 dvWdLQesv21a 0 \n",
"2019-08-05 15:52:41.009999990+03:00 dvWdLQesv21a 0 \n",
"2019-08-05 16:12:41.009999990+03:00 dvWdLQesv21a 0 \n",
"2019-08-05 16:32:41.009999990+03:00 dvWdLQesv21a 1 \n",
"2019-08-05 16:52:41.009999990+03:00 dvWdLQesv21a 0 \n",
"\n",
" screen_off_count screen_use_count \\\n",
"2019-08-05 15:32:41.009999990+03:00 0 0 \n",
"2019-08-05 15:52:41.009999990+03:00 0 0 \n",
"2019-08-05 16:12:41.009999990+03:00 0 0 \n",
"2019-08-05 16:32:41.009999990+03:00 0 0 \n",
"2019-08-05 16:52:41.009999990+03:00 0 0 \n",
"\n",
" device \n",
"2019-08-05 15:32:41.009999990+03:00 i8jmoIuoe12Mo \n",
"2019-08-05 15:52:41.009999990+03:00 i8jmoIuoe12Mo \n",
"2019-08-05 16:12:41.009999990+03:00 i8jmoIuoe12Mo \n",
"2019-08-05 16:32:41.009999990+03:00 i8jmoIuoe12Mo \n",
"2019-08-05 16:52:41.009999990+03:00 i8jmoIuoe12Mo "
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"empty_bat = pd.DataFrame()\n",
"no_bat = s.screen_count(\n",
" data,\n",
" screen_column_name = \"screen_status\",\n",
" resample_args = {\"rule\":\"20T\",\"origin\":start_time}\n",
") #no battery information\n",
"no_bat.head()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "cedb46a6",
"metadata": {},
"source": [
"### 4.2 Extract features using the wrapper\n",
"We can use `niimpy`'s ready-made wrapper to extract one or several features at the same time. The wrapper will require two inputs:\n",
"- (mandatory) dataframe that must comply with the minimum requirements (see '* TIP! Data requirements above)\n",
"- (optional) an argument dictionary for wrapper\n",
"\n",
"#### 4.2.1 The argument dictionary for wrapper (or how we specify the way the wrapper works)\n",
"The argument dictionary contains the arguments for each stand-alone function we would like to employ. Its keys are the feature functions we want to compute. Its values are argument dictionaries created for each stand-alone function we will employ. \n",
"Let's see some examples of wrapper dictionaries:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "eb9e4e80",
"metadata": {},
"outputs": [],
"source": [
"wrapper_features1 = {s.screen_count:{\"screen_column_name\":\"screen_status\",\"resample_args\":{\"rule\":\"1D\"}},\n",
" s.screen_duration_min:{\"screen_column_name\":\"screen_status\",\"resample_args\":{\"rule\":\"1D\"}}}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ae73b44f",
"metadata": {},
"source": [
"- `wrapper_features1` will be used to analyze two features, `screen_count` and `screen_duration_min`. For the feature screen_count, we will use the data stored in the column `screen_status` in our dataframe and the data will be binned in one day periods. For the feature screen_duration_min, we will use the data stored in the column `screen_status` in our dataframe and the data will be binned in one day periods. "
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "0762b7ff",
"metadata": {},
"outputs": [],
"source": [
"wrapper_features2 = {s.screen_count:{\"screen_column_name\":\"screen_status\", \"battery_column_name\":\"battery_status\", \"resample_args\":{\"rule\":\"1D\"}},\n",
" s.screen_duration:{\"screen_column_name\":\"random_name\",\"resample_args\":{\"rule\":\"5H\",\"offset\":\"5min\"}}}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b166633f",
"metadata": {},
"source": [
"- `wrapper_features2` will be used to analyze two features, `screen_status` and `screen_duration`. For the feature screen_status, we will use the data stored in the column `screen_status` in our dataframe and the data will be binned in one day periods. In addition, we will use battery data stored in a column called \"battery_status\". For the feature screen_duration, we will use the data stored in the column `random_name` in our dataframe and the data will be binned in 5-hour periods with a 5-minute offset. "
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "e4539ae0",
"metadata": {},
"outputs": [],
"source": [
"wrapper_features3 = {s.screen_count:{\"screen_column_name\":\"one_name\",\"resample_args\":{\"rule\":\"1D\",\"offset\":\"5min\"}},\n",
" s.screen_duration:{\"screen_column_name\":\"one_name\", \"battery_column\":\"some_column\",\"resample_args\":{}},\n",
" s.screen_duration_min:{\"screen_column_name\":\"another_name\",\"resample_args\":{\"rule\":\"30T\",\"origin\":\"end_day\"}}}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "29e60a6f",
"metadata": {},
"source": [
"- `wrapper_features3` will be used to analyze three features, `screen_count`, `screen_duration`, and `screen_duration_min`. For the feature screen_count, we will use the data stored in the column `one_name` and the data will be binned in one day periods with a 5-min offset. For the feature screen_duration, we will use the data stored in the column `one_name` in our dataframe and the data will be binned using the default settings, i.e. 30-min bins. In addition, we will use data from the battery sensor, which will be passed in a column called \"some_column\". Finally, for the feature screen_duration_min, we will use the data stored in the column `another_name` in our dataframe and the data will be binned in 30-minute periods and the origin of the bins will be the ceiling midnight of the last day.\n",
"\n",
"**Default values:** if no arguments are passed, `niimpy`'s default values are \"screen_status\" for the screen_column_name, and 30-min aggregation bins. Moreover, the wrapper will compute all the available functions in absence of the argument dictionary. \n",
"\n",
"#### 4.2.2 Using the wrapper\n",
"Now that we understand how the wrapper is customized, it is time we compute our first communication feature using the wrapper. Suppose that we are interested in extracting the call total duration every 50 minutes. We will need `niimpy`'s `extract_features_comms` function, the data, and we will also need to create a dictionary to customize our function. Let's create the dictionary first"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "46ad73f5",
"metadata": {},
"outputs": [],
"source": [
"wrapper_features1 = {s.screen_duration:{\"screen_column_name\":\"screen_status\",\"resample_args\":{\"rule\":\"50T\"}}}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "5f6f8356",
"metadata": {},
"source": [
"Now, let's use the wrapper"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "0663affa",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" device | \n",
" screen_off_durationtotal | \n",
" screen_use_durationtotal | \n",
" screen_on_durationtotal | \n",
"
\n",
" \n",
" \n",
" \n",
" 2019-08-05 15:50:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" NaN | \n",
" NaN | \n",
" 3000.0 | \n",
"
\n",
" \n",
" 2019-08-05 16:40:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 3000.0 | \n",
" NaN | \n",
" 0.0 | \n",
"
\n",
" \n",
" 2019-08-05 17:30:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 0.0 | \n",
" 3000.0 | \n",
" 0.0 | \n",
"
\n",
" \n",
" 2019-08-05 18:20:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 0.0 | \n",
" 0.0 | \n",
" 0.0 | \n",
"
\n",
" \n",
" 2019-08-05 19:10:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 0.0 | \n",
" 0.0 | \n",
" 3000.0 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user device \\\n",
"2019-08-05 15:50:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 16:40:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 17:30:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 18:20:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 19:10:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"\n",
" screen_off_durationtotal screen_use_durationtotal \\\n",
"2019-08-05 15:50:00+03:00 NaN NaN \n",
"2019-08-05 16:40:00+03:00 3000.0 NaN \n",
"2019-08-05 17:30:00+03:00 0.0 3000.0 \n",
"2019-08-05 18:20:00+03:00 0.0 0.0 \n",
"2019-08-05 19:10:00+03:00 0.0 0.0 \n",
"\n",
" screen_on_durationtotal \n",
"2019-08-05 15:50:00+03:00 3000.0 \n",
"2019-08-05 16:40:00+03:00 0.0 \n",
"2019-08-05 17:30:00+03:00 0.0 \n",
"2019-08-05 18:20:00+03:00 0.0 \n",
"2019-08-05 19:10:00+03:00 3000.0 "
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"results_wrapper = s.extract_features_screen(data, bat_data, features=wrapper_features1)\n",
"results_wrapper.head(5)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "4be6a39e",
"metadata": {},
"source": [
"Our first attempt was succesful. Now, let's try something more. Let's assume we want to compute the screen_duration and screen_count in 50-minutes bin."
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "360ac232",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" device | \n",
" screen_off_durationtotal | \n",
" screen_use_durationtotal | \n",
" screen_on_durationtotal | \n",
" screen_on_count | \n",
" screen_off_count | \n",
" screen_use_count | \n",
"
\n",
" \n",
" \n",
" \n",
" 2019-08-05 15:50:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" NaN | \n",
" NaN | \n",
" 3000.0 | \n",
" 1 | \n",
" 0 | \n",
" 0 | \n",
"
\n",
" \n",
" 2019-08-05 16:40:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 3000.0 | \n",
" NaN | \n",
" 0.0 | \n",
" 0 | \n",
" 1 | \n",
" 0 | \n",
"
\n",
" \n",
" 2019-08-05 17:30:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 0.0 | \n",
" 3000.0 | \n",
" 0.0 | \n",
" 0 | \n",
" 0 | \n",
" 1 | \n",
"
\n",
" \n",
" 2019-08-05 18:20:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 0.0 | \n",
" 0.0 | \n",
" 0.0 | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
"
\n",
" \n",
" 2019-08-05 19:10:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 0.0 | \n",
" 0.0 | \n",
" 3000.0 | \n",
" 1 | \n",
" 0 | \n",
" 0 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user device \\\n",
"2019-08-05 15:50:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 16:40:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 17:30:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 18:20:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 19:10:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"\n",
" screen_off_durationtotal screen_use_durationtotal \\\n",
"2019-08-05 15:50:00+03:00 NaN NaN \n",
"2019-08-05 16:40:00+03:00 3000.0 NaN \n",
"2019-08-05 17:30:00+03:00 0.0 3000.0 \n",
"2019-08-05 18:20:00+03:00 0.0 0.0 \n",
"2019-08-05 19:10:00+03:00 0.0 0.0 \n",
"\n",
" screen_on_durationtotal screen_on_count \\\n",
"2019-08-05 15:50:00+03:00 3000.0 1 \n",
"2019-08-05 16:40:00+03:00 0.0 0 \n",
"2019-08-05 17:30:00+03:00 0.0 0 \n",
"2019-08-05 18:20:00+03:00 0.0 0 \n",
"2019-08-05 19:10:00+03:00 3000.0 1 \n",
"\n",
" screen_off_count screen_use_count \n",
"2019-08-05 15:50:00+03:00 0 0 \n",
"2019-08-05 16:40:00+03:00 1 0 \n",
"2019-08-05 17:30:00+03:00 0 1 \n",
"2019-08-05 18:20:00+03:00 0 0 \n",
"2019-08-05 19:10:00+03:00 0 0 "
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"wrapper_features2 = {s.screen_duration:{\"screen_column_name\":\"screen_status\",\"resample_args\":{\"rule\":\"50T\"}},\n",
" s.screen_count:{\"screen_column_name\":\"screen_status\",\"resample_args\":{\"rule\":\"50T\"}}}\n",
"results_wrapper = s.extract_features_screen(data, bat_data, features=wrapper_features2)\n",
"results_wrapper.head(5)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "6bb4f84b",
"metadata": {},
"source": [
"Great! Another successful attempt. We see from the results that more columns were added with the required calculations. This is how the wrapper works when all features are computed with the same bins. Now, let's see how the wrapper performs when each function has different binning requirements. Let's assume we need to compute the screen_duration every day, and the screen_count every 5 hours with an offset of 5 minutes."
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "91107404",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" device | \n",
" screen_off_durationtotal | \n",
" screen_use_durationtotal | \n",
" screen_on_durationtotal | \n",
" screen_on_count | \n",
" screen_off_count | \n",
" screen_use_count | \n",
"
\n",
" \n",
" \n",
" \n",
" 2019-08-05 00:00:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 6000.000000 | \n",
" 6000.000 | \n",
" 9000.000 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
" 2019-08-06 00:00:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 24000.000000 | \n",
" 15000.000 | \n",
" 27000.000 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
" 2019-08-07 00:00:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 15000.000000 | \n",
" 15000.000 | \n",
" 21000.000 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
" 2019-08-05 00:00:00+03:00 | \n",
" iGyXetHE3S8u | \n",
" Cq9vueHh3zVs | \n",
" 287266.927999 | \n",
" 1.189 | \n",
" 276.382 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
" 2019-08-06 00:00:00+03:00 | \n",
" iGyXetHE3S8u | \n",
" Cq9vueHh3zVs | \n",
" 0.000000 | \n",
" 0.000 | \n",
" 0.000 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user device \\\n",
"2019-08-05 00:00:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-06 00:00:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-07 00:00:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 00:00:00+03:00 iGyXetHE3S8u Cq9vueHh3zVs \n",
"2019-08-06 00:00:00+03:00 iGyXetHE3S8u Cq9vueHh3zVs \n",
"\n",
" screen_off_durationtotal screen_use_durationtotal \\\n",
"2019-08-05 00:00:00+03:00 6000.000000 6000.000 \n",
"2019-08-06 00:00:00+03:00 24000.000000 15000.000 \n",
"2019-08-07 00:00:00+03:00 15000.000000 15000.000 \n",
"2019-08-05 00:00:00+03:00 287266.927999 1.189 \n",
"2019-08-06 00:00:00+03:00 0.000000 0.000 \n",
"\n",
" screen_on_durationtotal screen_on_count \\\n",
"2019-08-05 00:00:00+03:00 9000.000 NaN \n",
"2019-08-06 00:00:00+03:00 27000.000 NaN \n",
"2019-08-07 00:00:00+03:00 21000.000 NaN \n",
"2019-08-05 00:00:00+03:00 276.382 NaN \n",
"2019-08-06 00:00:00+03:00 0.000 NaN \n",
"\n",
" screen_off_count screen_use_count \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2019-08-06 00:00:00+03:00 NaN NaN \n",
"2019-08-07 00:00:00+03:00 NaN NaN \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2019-08-06 00:00:00+03:00 NaN NaN "
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"wrapper_features3 = {s.screen_duration:{\"screen_column_name\":\"screen_status\",\"resample_args\":{\"rule\":\"1D\"}},\n",
" s.screen_count:{\"screen_column_name\":\"screen_status\",\"resample_args\":{\"rule\":\"5H\",\"offset\":\"5min\"}}}\n",
"results_wrapper = s.extract_features_screen(data, bat_data, features=wrapper_features3)\n",
"results_wrapper.head(5)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "85e5de98",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" device | \n",
" screen_off_durationtotal | \n",
" screen_use_durationtotal | \n",
" screen_on_durationtotal | \n",
" screen_on_count | \n",
" screen_off_count | \n",
" screen_use_count | \n",
"
\n",
" \n",
" \n",
" \n",
" 2020-01-09 00:05:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" 7.0 | \n",
" 8.0 | \n",
" 5.0 | \n",
"
\n",
" \n",
" 2020-01-09 05:05:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" 0.0 | \n",
" 0.0 | \n",
" 0.0 | \n",
"
\n",
" \n",
" 2020-01-09 10:05:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" 9.0 | \n",
" 9.0 | \n",
" 3.0 | \n",
"
\n",
" \n",
" 2020-01-09 15:05:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" 17.0 | \n",
" 17.0 | \n",
" 7.0 | \n",
"
\n",
" \n",
" 2020-01-09 20:05:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" 12.0 | \n",
" 11.0 | \n",
" 3.0 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user device \\\n",
"2020-01-09 00:05:00+02:00 jd9INuQ5BBlW OWd1Uau8POix \n",
"2020-01-09 05:05:00+02:00 jd9INuQ5BBlW OWd1Uau8POix \n",
"2020-01-09 10:05:00+02:00 jd9INuQ5BBlW OWd1Uau8POix \n",
"2020-01-09 15:05:00+02:00 jd9INuQ5BBlW OWd1Uau8POix \n",
"2020-01-09 20:05:00+02:00 jd9INuQ5BBlW OWd1Uau8POix \n",
"\n",
" screen_off_durationtotal screen_use_durationtotal \\\n",
"2020-01-09 00:05:00+02:00 NaN NaN \n",
"2020-01-09 05:05:00+02:00 NaN NaN \n",
"2020-01-09 10:05:00+02:00 NaN NaN \n",
"2020-01-09 15:05:00+02:00 NaN NaN \n",
"2020-01-09 20:05:00+02:00 NaN NaN \n",
"\n",
" screen_on_durationtotal screen_on_count \\\n",
"2020-01-09 00:05:00+02:00 NaN 7.0 \n",
"2020-01-09 05:05:00+02:00 NaN 0.0 \n",
"2020-01-09 10:05:00+02:00 NaN 9.0 \n",
"2020-01-09 15:05:00+02:00 NaN 17.0 \n",
"2020-01-09 20:05:00+02:00 NaN 12.0 \n",
"\n",
" screen_off_count screen_use_count \n",
"2020-01-09 00:05:00+02:00 8.0 5.0 \n",
"2020-01-09 05:05:00+02:00 0.0 0.0 \n",
"2020-01-09 10:05:00+02:00 9.0 3.0 \n",
"2020-01-09 15:05:00+02:00 17.0 7.0 \n",
"2020-01-09 20:05:00+02:00 11.0 3.0 "
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"results_wrapper.tail(5)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e76734c7",
"metadata": {},
"source": [
"The output is once again a dataframe. In this case, two aggregations are shown. The first one is the daily aggregation computed for the `screen_duration` feature (head). The second one is the 5-hour aggregation period with 5-min offset for the `screen_count` (tail). We must note that because the `screen_count`feature is not required to be aggregated daily, the daily aggregation timestamps have a NaN value. Similarly, because the `screen_duration`is not required to be aggregated in 5-hour windows, its values are NaN for all subjects. "
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "358f307a",
"metadata": {},
"source": [
"#### 4.2.3 Wrapper and its default option\n",
"The default option will compute all features in 30-minute aggregation windows. To use the `extract_features_comms` function with its default options, simply call the function. "
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "fcb09628",
"metadata": {},
"outputs": [],
"source": [
"default = s.extract_features_screen(data, bat_data)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "949adeca",
"metadata": {},
"source": [
"The function prints the computed features so you can track its process. Now let's have a look at the outputs"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "c0fa2718",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" device | \n",
" screen_on_count | \n",
" screen_off_count | \n",
" screen_use_count | \n",
" screen_off_durationtotal | \n",
" screen_use_durationtotal | \n",
" screen_on_durationtotal | \n",
" screen_off_durationminimum | \n",
" screen_on_durationminimum | \n",
" ... | \n",
" screen_off_durationmean | \n",
" screen_on_durationmean | \n",
" screen_use_durationmean | \n",
" screen_on_durationmedian | \n",
" screen_use_durationmedian | \n",
" screen_off_durationmedian | \n",
" screen_on_durationstd | \n",
" screen_use_durationstd | \n",
" screen_off_durationstd | \n",
" first_unlock | \n",
"
\n",
" \n",
" \n",
" \n",
" 2020-01-09 20:00:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 0.0 | \n",
" 0.0 | \n",
" 0.0 | \n",
" 0.000 | \n",
" 0.000 | \n",
" 0.000 | \n",
" NaN | \n",
" NaN | \n",
" ... | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaT | \n",
"
\n",
" \n",
" 2020-01-09 20:30:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.0 | \n",
" 1.0 | \n",
" 1.0 | \n",
" 0.005 | \n",
" 28.930 | \n",
" 8.253 | \n",
" 0.005 | \n",
" 8.253 | \n",
" ... | \n",
" 0.005000 | \n",
" 8.253000 | \n",
" 28.930 | \n",
" 8.2530 | \n",
" 28.930 | \n",
" 0.005 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaT | \n",
"
\n",
" \n",
" 2020-01-09 21:00:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 2.0 | \n",
" 1.0 | \n",
" 1.0 | \n",
" 0.010 | \n",
" 39.087 | \n",
" 11.158 | \n",
" 0.010 | \n",
" 5.234 | \n",
" ... | \n",
" 0.010000 | \n",
" 5.579000 | \n",
" 39.087 | \n",
" 5.5790 | \n",
" 39.087 | \n",
" 0.010 | \n",
" 0.487904 | \n",
" NaN | \n",
" NaN | \n",
" NaT | \n",
"
\n",
" \n",
" 2020-01-09 21:30:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 4.0 | \n",
" 5.0 | \n",
" 1.0 | \n",
" 46.028 | \n",
" 101.062 | \n",
" 376.930 | \n",
" 0.006 | \n",
" 33.834 | \n",
" ... | \n",
" 9.205600 | \n",
" 94.232500 | \n",
" 101.062 | \n",
" 73.2835 | \n",
" 101.062 | \n",
" 0.012 | \n",
" 71.990324 | \n",
" NaN | \n",
" 20.561987 | \n",
" NaT | \n",
"
\n",
" \n",
" 2020-01-09 22:00:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 1.0 | \n",
" 1.0 | \n",
" 0.0 | \n",
" 0.011 | \n",
" NaN | \n",
" 154.643 | \n",
" 0.011 | \n",
" 154.643 | \n",
" ... | \n",
" 0.011000 | \n",
" 154.643000 | \n",
" NaN | \n",
" 154.6430 | \n",
" NaN | \n",
" 0.011 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaT | \n",
"
\n",
" \n",
" 2020-01-09 22:30:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 0.0 | \n",
" 0.0 | \n",
" 0.0 | \n",
" 0.000 | \n",
" NaN | \n",
" 0.000 | \n",
" NaN | \n",
" NaN | \n",
" ... | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaT | \n",
"
\n",
" \n",
" 2020-01-09 23:00:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" 4.0 | \n",
" 3.0 | \n",
" 0.0 | \n",
" 0.025 | \n",
" NaN | \n",
" 6.931 | \n",
" 0.008 | \n",
" 2.079 | \n",
" ... | \n",
" 0.008333 | \n",
" 2.310333 | \n",
" NaN | \n",
" 2.2620 | \n",
" NaN | \n",
" 0.008 | \n",
" 0.258906 | \n",
" NaN | \n",
" 0.000577 | \n",
" NaT | \n",
"
\n",
" \n",
" 2019-08-05 00:00:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" ... | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" 2019-08-05 16:32:41.009999990+03:00 | \n",
"
\n",
" \n",
" 2019-08-05 00:00:00+03:00 | \n",
" iGyXetHE3S8u | \n",
" Cq9vueHh3zVs | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" ... | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" 2019-08-05 14:03:42.322000027+03:00 | \n",
"
\n",
" \n",
" 2020-01-09 00:00:00+02:00 | \n",
" jd9INuQ5BBlW | \n",
" OWd1Uau8POix | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" ... | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" 2020-01-09 02:16:19.010999918+02:00 | \n",
"
\n",
" \n",
"
\n",
"
10 rows × 24 columns
\n",
"
"
],
"text/plain": [
" user device screen_on_count \\\n",
"2020-01-09 20:00:00+02:00 jd9INuQ5BBlW OWd1Uau8POix 0.0 \n",
"2020-01-09 20:30:00+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.0 \n",
"2020-01-09 21:00:00+02:00 jd9INuQ5BBlW OWd1Uau8POix 2.0 \n",
"2020-01-09 21:30:00+02:00 jd9INuQ5BBlW OWd1Uau8POix 4.0 \n",
"2020-01-09 22:00:00+02:00 jd9INuQ5BBlW OWd1Uau8POix 1.0 \n",
"2020-01-09 22:30:00+02:00 jd9INuQ5BBlW OWd1Uau8POix 0.0 \n",
"2020-01-09 23:00:00+02:00 jd9INuQ5BBlW OWd1Uau8POix 4.0 \n",
"2019-08-05 00:00:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo NaN \n",
"2019-08-05 00:00:00+03:00 iGyXetHE3S8u Cq9vueHh3zVs NaN \n",
"2020-01-09 00:00:00+02:00 jd9INuQ5BBlW OWd1Uau8POix NaN \n",
"\n",
" screen_off_count screen_use_count \\\n",
"2020-01-09 20:00:00+02:00 0.0 0.0 \n",
"2020-01-09 20:30:00+02:00 1.0 1.0 \n",
"2020-01-09 21:00:00+02:00 1.0 1.0 \n",
"2020-01-09 21:30:00+02:00 5.0 1.0 \n",
"2020-01-09 22:00:00+02:00 1.0 0.0 \n",
"2020-01-09 22:30:00+02:00 0.0 0.0 \n",
"2020-01-09 23:00:00+02:00 3.0 0.0 \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2020-01-09 00:00:00+02:00 NaN NaN \n",
"\n",
" screen_off_durationtotal screen_use_durationtotal \\\n",
"2020-01-09 20:00:00+02:00 0.000 0.000 \n",
"2020-01-09 20:30:00+02:00 0.005 28.930 \n",
"2020-01-09 21:00:00+02:00 0.010 39.087 \n",
"2020-01-09 21:30:00+02:00 46.028 101.062 \n",
"2020-01-09 22:00:00+02:00 0.011 NaN \n",
"2020-01-09 22:30:00+02:00 0.000 NaN \n",
"2020-01-09 23:00:00+02:00 0.025 NaN \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2020-01-09 00:00:00+02:00 NaN NaN \n",
"\n",
" screen_on_durationtotal \\\n",
"2020-01-09 20:00:00+02:00 0.000 \n",
"2020-01-09 20:30:00+02:00 8.253 \n",
"2020-01-09 21:00:00+02:00 11.158 \n",
"2020-01-09 21:30:00+02:00 376.930 \n",
"2020-01-09 22:00:00+02:00 154.643 \n",
"2020-01-09 22:30:00+02:00 0.000 \n",
"2020-01-09 23:00:00+02:00 6.931 \n",
"2019-08-05 00:00:00+03:00 NaN \n",
"2019-08-05 00:00:00+03:00 NaN \n",
"2020-01-09 00:00:00+02:00 NaN \n",
"\n",
" screen_off_durationminimum \\\n",
"2020-01-09 20:00:00+02:00 NaN \n",
"2020-01-09 20:30:00+02:00 0.005 \n",
"2020-01-09 21:00:00+02:00 0.010 \n",
"2020-01-09 21:30:00+02:00 0.006 \n",
"2020-01-09 22:00:00+02:00 0.011 \n",
"2020-01-09 22:30:00+02:00 NaN \n",
"2020-01-09 23:00:00+02:00 0.008 \n",
"2019-08-05 00:00:00+03:00 NaN \n",
"2019-08-05 00:00:00+03:00 NaN \n",
"2020-01-09 00:00:00+02:00 NaN \n",
"\n",
" screen_on_durationminimum ... \\\n",
"2020-01-09 20:00:00+02:00 NaN ... \n",
"2020-01-09 20:30:00+02:00 8.253 ... \n",
"2020-01-09 21:00:00+02:00 5.234 ... \n",
"2020-01-09 21:30:00+02:00 33.834 ... \n",
"2020-01-09 22:00:00+02:00 154.643 ... \n",
"2020-01-09 22:30:00+02:00 NaN ... \n",
"2020-01-09 23:00:00+02:00 2.079 ... \n",
"2019-08-05 00:00:00+03:00 NaN ... \n",
"2019-08-05 00:00:00+03:00 NaN ... \n",
"2020-01-09 00:00:00+02:00 NaN ... \n",
"\n",
" screen_off_durationmean screen_on_durationmean \\\n",
"2020-01-09 20:00:00+02:00 NaN NaN \n",
"2020-01-09 20:30:00+02:00 0.005000 8.253000 \n",
"2020-01-09 21:00:00+02:00 0.010000 5.579000 \n",
"2020-01-09 21:30:00+02:00 9.205600 94.232500 \n",
"2020-01-09 22:00:00+02:00 0.011000 154.643000 \n",
"2020-01-09 22:30:00+02:00 NaN NaN \n",
"2020-01-09 23:00:00+02:00 0.008333 2.310333 \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2020-01-09 00:00:00+02:00 NaN NaN \n",
"\n",
" screen_use_durationmean screen_on_durationmedian \\\n",
"2020-01-09 20:00:00+02:00 NaN NaN \n",
"2020-01-09 20:30:00+02:00 28.930 8.2530 \n",
"2020-01-09 21:00:00+02:00 39.087 5.5790 \n",
"2020-01-09 21:30:00+02:00 101.062 73.2835 \n",
"2020-01-09 22:00:00+02:00 NaN 154.6430 \n",
"2020-01-09 22:30:00+02:00 NaN NaN \n",
"2020-01-09 23:00:00+02:00 NaN 2.2620 \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2020-01-09 00:00:00+02:00 NaN NaN \n",
"\n",
" screen_use_durationmedian \\\n",
"2020-01-09 20:00:00+02:00 NaN \n",
"2020-01-09 20:30:00+02:00 28.930 \n",
"2020-01-09 21:00:00+02:00 39.087 \n",
"2020-01-09 21:30:00+02:00 101.062 \n",
"2020-01-09 22:00:00+02:00 NaN \n",
"2020-01-09 22:30:00+02:00 NaN \n",
"2020-01-09 23:00:00+02:00 NaN \n",
"2019-08-05 00:00:00+03:00 NaN \n",
"2019-08-05 00:00:00+03:00 NaN \n",
"2020-01-09 00:00:00+02:00 NaN \n",
"\n",
" screen_off_durationmedian screen_on_durationstd \\\n",
"2020-01-09 20:00:00+02:00 NaN NaN \n",
"2020-01-09 20:30:00+02:00 0.005 NaN \n",
"2020-01-09 21:00:00+02:00 0.010 0.487904 \n",
"2020-01-09 21:30:00+02:00 0.012 71.990324 \n",
"2020-01-09 22:00:00+02:00 0.011 NaN \n",
"2020-01-09 22:30:00+02:00 NaN NaN \n",
"2020-01-09 23:00:00+02:00 0.008 0.258906 \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2020-01-09 00:00:00+02:00 NaN NaN \n",
"\n",
" screen_use_durationstd screen_off_durationstd \\\n",
"2020-01-09 20:00:00+02:00 NaN NaN \n",
"2020-01-09 20:30:00+02:00 NaN NaN \n",
"2020-01-09 21:00:00+02:00 NaN NaN \n",
"2020-01-09 21:30:00+02:00 NaN 20.561987 \n",
"2020-01-09 22:00:00+02:00 NaN NaN \n",
"2020-01-09 22:30:00+02:00 NaN NaN \n",
"2020-01-09 23:00:00+02:00 NaN 0.000577 \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2019-08-05 00:00:00+03:00 NaN NaN \n",
"2020-01-09 00:00:00+02:00 NaN NaN \n",
"\n",
" first_unlock \n",
"2020-01-09 20:00:00+02:00 NaT \n",
"2020-01-09 20:30:00+02:00 NaT \n",
"2020-01-09 21:00:00+02:00 NaT \n",
"2020-01-09 21:30:00+02:00 NaT \n",
"2020-01-09 22:00:00+02:00 NaT \n",
"2020-01-09 22:30:00+02:00 NaT \n",
"2020-01-09 23:00:00+02:00 NaT \n",
"2019-08-05 00:00:00+03:00 2019-08-05 16:32:41.009999990+03:00 \n",
"2019-08-05 00:00:00+03:00 2019-08-05 14:03:42.322000027+03:00 \n",
"2020-01-09 00:00:00+02:00 2020-01-09 02:16:19.010999918+02:00 \n",
"\n",
"[10 rows x 24 columns]"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"default.tail(10)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "3d6ddcf4",
"metadata": {},
"source": [
"## Implementing own features"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "bc43923f",
"metadata": {},
"source": [
"If none of the provided functions suits well, We can implement our own customized features easily. To do so, we need to define a function that accepts a dataframe and returns a dataframe. The returned object should be indexed by user and timestamps (multiindex).\n",
"Let's assume we need a new function that detects the last time the screen is unlocked. Let's first define the function"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "e36a48ed",
"metadata": {},
"outputs": [],
"source": [
"def screen_last_unlock(df, bat, screen_column_name = \"screen_status\", resample_args = {\"rule\":\"30T\"}): \n",
" df2 = s.util_screen(df, bat, screen_column_name=screen_column_name)\n",
" df2 = s.event_classification_screen(df2, screen_column_name=screen_column_name)\n",
" \n",
" df2[\"time\"] = df2.index\n",
" result = df2[df2.on==1].groupby([\"user\", \"device\"])[\"time\"].resample(**resample_args).max()\n",
" result = result.to_frame(name=\"first_unlock\")\n",
" result = result.reset_index([\"user\", \"device\"])\n",
" \n",
" return result"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "1f86b979",
"metadata": {},
"source": [
"Then, we can call our new function in the stand-alone way or using the `extract_features_screen` function. Because the stand-alone way is the common way to call functions in python, we will not show it. Instead, we will show how to integrate this new function to the wrapper. Let's read again the data and assume we want the default behavior of the wrapper. "
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "0ee78bf7",
"metadata": {},
"outputs": [],
"source": [
"customized_features = s.extract_features_screen(data, bat_data, features={screen_last_unlock: {}})"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "babdda7a",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" user | \n",
" device | \n",
" first_unlock | \n",
"
\n",
" \n",
" \n",
" \n",
" 2019-08-05 16:30:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" 2019-08-05 16:32:41.009999990+03:00 | \n",
"
\n",
" \n",
" 2019-08-05 17:00:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" NaT | \n",
"
\n",
" \n",
" 2019-08-05 17:30:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" NaT | \n",
"
\n",
" \n",
" 2019-08-05 18:00:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" NaT | \n",
"
\n",
" \n",
" 2019-08-05 18:30:00+03:00 | \n",
" dvWdLQesv21a | \n",
" i8jmoIuoe12Mo | \n",
" NaT | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" user device \\\n",
"2019-08-05 16:30:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 17:00:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 17:30:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 18:00:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"2019-08-05 18:30:00+03:00 dvWdLQesv21a i8jmoIuoe12Mo \n",
"\n",
" first_unlock \n",
"2019-08-05 16:30:00+03:00 2019-08-05 16:32:41.009999990+03:00 \n",
"2019-08-05 17:00:00+03:00 NaT \n",
"2019-08-05 17:30:00+03:00 NaT \n",
"2019-08-05 18:00:00+03:00 NaT \n",
"2019-08-05 18:30:00+03:00 NaT "
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"customized_features.head()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "niimpy",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}