Predictive-Maintenance-Data-Preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "365ed4bc",
   "metadata": {},
   "source": [
    "# Predictive maintenance\n",
    "\n",
    "## Part 1: Data Preparation\n",
    "\n",
    "The original data can be [downloaded from this link.](https://ti.arc.nasa.gov/tech/dash/groups/pcoe/prognostic-data-repository/#turbofan) Since the content in the train and test datasets is different, we are making it uniform before we start the data exploration and the model buiding process. We will convert the data into a more natural format for Vaex."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b7f353ac",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-09-02T09:26:28.899712Z",
     "start_time": "2021-09-02T09:26:28.251469Z"
    }
   },
   "outputs": [],
   "source": [
    "import vaex"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e878eb1",
   "metadata": {},
   "source": [
    "### Read the data\n",
    "\n",
    "The data contains a list of sensors. These are their names and meanings:\n",
    "\n",
    "\n",
    "| Name      |Description                      |Unit     |    \n",
    "|-----------|---------------------------------|---------|    \n",
    "| T2        | Total temperature at fan inlet  | °R      |    \n",
    "| T24       | Total temperature at LPC outlet | °R      |    \n",
    "| T30       | Total temperature at HPC outlet | °R      |    \n",
    "| T50       | Total temperature at LPT outlet | °R      |    \n",
    "| P2        | Pressure at fan inlet           | psia    |    \n",
    "| P15       | Total pressure in bypass-duct   | psia    |    \n",
    "| P30       | Total pressure at HPC outlet    | psia    |    \n",
    "| Nf        | Physical fan speed              | rpm     |    \n",
    "| Nc        | Physical core speed             | rpm     |    \n",
    "| epr       | Engine pressure ratio (P50/P2)  | --      |    \n",
    "| Ps30      | Static pressure at HPC outlet   | psia    |    \n",
    "| phi       | Ratio of fuel flow to Ps30      | pps/psi |    \n",
    "| NRf       | Corrected fan speed             | rpm     |    \n",
    "| NRc       | Corrected core speed            | rpm     |    \n",
    "| BPR       | Bypass Ratio                    | --      |    \n",
    "| farB      | Burner fuel-air ratio           | --      |    \n",
    "| htBleed   | Bleed Enthalpy                  | --      |    \n",
    "| Nf_dmd    | Demanded fan speed              | rpm     |    \n",
    "| PCNfR_dmd | Demanded corrected fan speed    | rpm     |    \n",
    "| W31       | HPT coolant bleed               | lbm/s   |    \n",
    "| W32       | LPT coolant bleed               | lbm/s   |    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cf4e19b2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-09-02T09:26:37.482537Z",
     "start_time": "2021-09-02T09:26:37.376971Z"
    }
   },
   "outputs": [],
   "source": [
    "column_names = ['unit_number', 'time_in_cycles', 'setting_1', 'setting_2', 'setting_3',\n",
    "                'T2', 'T24', 'T30', 'T50', 'P2', 'P15', 'P30', 'Nf', 'Nc', 'epr', 'Ps30', 'phi', \n",
    "                'NRf', 'NRc', 'BPR', 'farB', 'htBleed', 'Nf_dmd', 'PCNfR_dmd', 'W31', 'W32']\n",
    "\n",
    "\n",
    "# The training data\n",
    "train_data = vaex.read_csv(\"./data/train_FD001.txt\", sep='\\s+', names=column_names)\n",
    "\n",
    "# The testing data\n",
    "test_data = vaex.read_csv(\"./data/test_FD001.txt\", sep='\\s+', names=column_names)\n",
    "\n",
    "# The \"answer\" to the test data\n",
    "y_test = vaex.read_csv('./data/RUL_FD001.txt', names=['remaining_cycles'])\n",
    "y_test['unit_number'] = vaex.vrange(1, 101)\n",
    "y_test['unit_number'] = y_test.unit_number.astype('int')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99e7917e",
   "metadata": {},
   "source": [
    "### Create proper train and test datasets\n",
    "\n",
    "- in the training set, the engines are run until failure occurs, so we can calculate the target varuable, i,e, the RUL (Remaining Useful Life) based on when a particular engines running;\n",
    "- in the test set the engines are run for some time, and our goal is to predict their RULs. Their RUL are provided in a separate file, so we need to join it so it can be made available for evaluating scores and estimateing model performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3da9cdc9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-09-02T09:26:39.567011Z",
     "start_time": "2021-09-02T09:26:39.562124Z"
    }
   },
   "outputs": [],
   "source": [
    "def prepare_data(data, y=None):\n",
    "    df = data.copy()  # As to not modify the underlying dataframe\n",
    "    # Count how many cycles each unit is run for - groupby and count\n",
    "    g = df.groupby('unit_number').agg({'max_cycles': vaex.agg.count('time_in_cycles')})\n",
    "    # Join to the main data - basically adds the \"max_cycle\" column\n",
    "    df = df.join(other=g, on='unit_number', how='left')\n",
    "    \n",
    "    # Calculate the RUL:\n",
    "    if y is None:  # This is for the train data -> last point is the point of failure\n",
    "        # Calculate the RUL\n",
    "        df['RUL'] = df.max_cycles - df.time_in_cycles\n",
    "        # Drop the column that is not needed anymore\n",
    "        df = df.drop(columns=['max_cycles'])\n",
    "    else:  # This is for the test data -> add the answer to calculate the RUL\n",
    "        # Join the answers\n",
    "        df = df.join(y, on='unit_number', how='left')\n",
    "        # Calculate the RUL\n",
    "        df['RUL'] = df.max_cycles + df.remaining_cycles - df.time_in_cycles\n",
    "        # Drop the columns that are not needed anymore\n",
    "        df = df.drop(columns=['remaining_cycles', 'max_cycles'])    \n",
    "    # Done\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ad6238c6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-09-02T09:26:42.127661Z",
     "start_time": "2021-09-02T09:26:42.029977Z"
    }
   },
   "outputs": [],
   "source": [
    "# Add the RUL to the train and test sets\n",
    "df_train = prepare_data(train_data)\n",
    "df_test = prepare_data(test_data, y=y_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db0d69ba",
   "metadata": {},
   "source": [
    "### Quick preview of the datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fb195827",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-09-02T09:26:44.164818Z",
     "start_time": "2021-09-02T09:26:44.126866Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>#                                 </th><th>unit_number  </th><th>time_in_cycles  </th><th>setting_1  </th><th>setting_2  </th><th>setting_3  </th><th>T2    </th><th>T24   </th><th>T30    </th><th>T50    </th><th>P2   </th><th>P15  </th><th>P30   </th><th>Nf     </th><th>Nc     </th><th>epr  </th><th>Ps30  </th><th>phi   </th><th>NRf    </th><th>NRc    </th><th>BPR   </th><th>farB  </th><th>htBleed  </th><th>Nf_dmd  </th><th>PCNfR_dmd  </th><th>W31  </th><th>W32    </th><th>RUL  </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td><i style='opacity: 0.6'>0</i>     </td><td>1            </td><td>1               </td><td>-0.0007    </td><td>-0.0004    </td><td>100.0      </td><td>518.67</td><td>641.82</td><td>1589.7 </td><td>1400.6 </td><td>14.62</td><td>21.61</td><td>554.36</td><td>2388.06</td><td>9046.19</td><td>1.3  </td><td>47.47 </td><td>521.66</td><td>2388.02</td><td>8138.62</td><td>8.4195</td><td>0.03  </td><td>392      </td><td>2388    </td><td>100.0      </td><td>39.06</td><td>23.419 </td><td>191  </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>1</i>     </td><td>1            </td><td>2               </td><td>0.0019     </td><td>-0.0003    </td><td>100.0      </td><td>518.67</td><td>642.15</td><td>1591.82</td><td>1403.14</td><td>14.62</td><td>21.61</td><td>553.75</td><td>2388.04</td><td>9044.07</td><td>1.3  </td><td>47.49 </td><td>522.28</td><td>2388.07</td><td>8131.49</td><td>8.4318</td><td>0.03  </td><td>392      </td><td>2388    </td><td>100.0      </td><td>39.0 </td><td>23.4236</td><td>190  </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>2</i>     </td><td>1            </td><td>3               </td><td>-0.0043    </td><td>0.0003     </td><td>100.0      </td><td>518.67</td><td>642.35</td><td>1587.99</td><td>1404.2 </td><td>14.62</td><td>21.61</td><td>554.26</td><td>2388.08</td><td>9052.94</td><td>1.3  </td><td>47.27 </td><td>522.42</td><td>2388.03</td><td>8133.23</td><td>8.4178</td><td>0.03  </td><td>390      </td><td>2388    </td><td>100.0      </td><td>38.95</td><td>23.3442</td><td>189  </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>3</i>     </td><td>1            </td><td>4               </td><td>0.0007     </td><td>0.0        </td><td>100.0      </td><td>518.67</td><td>642.35</td><td>1582.79</td><td>1401.87</td><td>14.62</td><td>21.61</td><td>554.45</td><td>2388.11</td><td>9049.48</td><td>1.3  </td><td>47.13 </td><td>522.86</td><td>2388.08</td><td>8133.83</td><td>8.3682</td><td>0.03  </td><td>392      </td><td>2388    </td><td>100.0      </td><td>38.88</td><td>23.3739</td><td>188  </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>4</i>     </td><td>1            </td><td>5               </td><td>-0.0019    </td><td>-0.0002    </td><td>100.0      </td><td>518.67</td><td>642.37</td><td>1582.85</td><td>1406.22</td><td>14.62</td><td>21.61</td><td>554.0 </td><td>2388.06</td><td>9055.15</td><td>1.3  </td><td>47.28 </td><td>522.19</td><td>2388.04</td><td>8133.8 </td><td>8.4294</td><td>0.03  </td><td>393      </td><td>2388    </td><td>100.0      </td><td>38.9 </td><td>23.4044</td><td>187  </td></tr>\n",
       "<tr><td>...                               </td><td>...          </td><td>...             </td><td>...        </td><td>...        </td><td>...        </td><td>...   </td><td>...   </td><td>...    </td><td>...    </td><td>...  </td><td>...  </td><td>...   </td><td>...    </td><td>...    </td><td>...  </td><td>...   </td><td>...   </td><td>...    </td><td>...    </td><td>...   </td><td>...   </td><td>...      </td><td>...     </td><td>...        </td><td>...  </td><td>...    </td><td>...  </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>20,626</i></td><td>100          </td><td>196             </td><td>-0.0004    </td><td>-0.0003    </td><td>100.0      </td><td>518.67</td><td>643.49</td><td>1597.98</td><td>1428.63</td><td>14.62</td><td>21.61</td><td>551.43</td><td>2388.19</td><td>9065.52</td><td>1.3  </td><td>48.07 </td><td>519.49</td><td>2388.26</td><td>8137.6 </td><td>8.4956</td><td>0.03  </td><td>397      </td><td>2388    </td><td>100.0      </td><td>38.49</td><td>22.9735</td><td>4    </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>20,627</i></td><td>100          </td><td>197             </td><td>-0.0016    </td><td>-0.0005    </td><td>100.0      </td><td>518.67</td><td>643.54</td><td>1604.5 </td><td>1433.58</td><td>14.62</td><td>21.61</td><td>550.86</td><td>2388.23</td><td>9065.11</td><td>1.3  </td><td>48.04 </td><td>519.68</td><td>2388.22</td><td>8136.5 </td><td>8.5139</td><td>0.03  </td><td>395      </td><td>2388    </td><td>100.0      </td><td>38.3 </td><td>23.1594</td><td>3    </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>20,628</i></td><td>100          </td><td>198             </td><td>0.0004     </td><td>0.0        </td><td>100.0      </td><td>518.67</td><td>643.42</td><td>1602.46</td><td>1428.18</td><td>14.62</td><td>21.61</td><td>550.94</td><td>2388.24</td><td>9065.9 </td><td>1.3  </td><td>48.09 </td><td>520.01</td><td>2388.24</td><td>8141.05</td><td>8.5646</td><td>0.03  </td><td>398      </td><td>2388    </td><td>100.0      </td><td>38.44</td><td>22.9333</td><td>2    </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>20,629</i></td><td>100          </td><td>199             </td><td>-0.0011    </td><td>0.0003     </td><td>100.0      </td><td>518.67</td><td>643.23</td><td>1605.26</td><td>1426.53</td><td>14.62</td><td>21.61</td><td>550.68</td><td>2388.25</td><td>9073.72</td><td>1.3  </td><td>48.39 </td><td>519.67</td><td>2388.23</td><td>8139.29</td><td>8.5389</td><td>0.03  </td><td>395      </td><td>2388    </td><td>100.0      </td><td>38.29</td><td>23.064 </td><td>1    </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>20,630</i></td><td>100          </td><td>200             </td><td>-0.0032    </td><td>-0.0005    </td><td>100.0      </td><td>518.67</td><td>643.85</td><td>1600.38</td><td>1432.14</td><td>14.62</td><td>21.61</td><td>550.79</td><td>2388.26</td><td>9061.48</td><td>1.3  </td><td>48.2  </td><td>519.3 </td><td>2388.26</td><td>8137.33</td><td>8.5036</td><td>0.03  </td><td>396      </td><td>2388    </td><td>100.0      </td><td>38.37</td><td>23.0522</td><td>0    </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "#       unit_number    time_in_cycles    setting_1    setting_2    setting_3    T2      T24     T30      T50      P2     P15    P30     Nf       Nc       epr    Ps30    phi     NRf      NRc      BPR     farB    htBleed    Nf_dmd    PCNfR_dmd    W31    W32      RUL\n",
       "0       1              1                 -0.0007      -0.0004      100.0        518.67  641.82  1589.7   1400.6   14.62  21.61  554.36  2388.06  9046.19  1.3    47.47   521.66  2388.02  8138.62  8.4195  0.03    392        2388      100.0        39.06  23.419   191\n",
       "1       1              2                 0.0019       -0.0003      100.0        518.67  642.15  1591.82  1403.14  14.62  21.61  553.75  2388.04  9044.07  1.3    47.49   522.28  2388.07  8131.49  8.4318  0.03    392        2388      100.0        39.0   23.4236  190\n",
       "2       1              3                 -0.0043      0.0003       100.0        518.67  642.35  1587.99  1404.2   14.62  21.61  554.26  2388.08  9052.94  1.3    47.27   522.42  2388.03  8133.23  8.4178  0.03    390        2388      100.0        38.95  23.3442  189\n",
       "3       1              4                 0.0007       0.0          100.0        518.67  642.35  1582.79  1401.87  14.62  21.61  554.45  2388.11  9049.48  1.3    47.13   522.86  2388.08  8133.83  8.3682  0.03    392        2388      100.0        38.88  23.3739  188\n",
       "4       1              5                 -0.0019      -0.0002      100.0        518.67  642.37  1582.85  1406.22  14.62  21.61  554.0   2388.06  9055.15  1.3    47.28   522.19  2388.04  8133.8   8.4294  0.03    393        2388      100.0        38.9   23.4044  187\n",
       "...     ...            ...               ...          ...          ...          ...     ...     ...      ...      ...    ...    ...     ...      ...      ...    ...     ...     ...      ...      ...     ...     ...        ...       ...          ...    ...      ...\n",
       "20,626  100            196               -0.0004      -0.0003      100.0        518.67  643.49  1597.98  1428.63  14.62  21.61  551.43  2388.19  9065.52  1.3    48.07   519.49  2388.26  8137.6   8.4956  0.03    397        2388      100.0        38.49  22.9735  4\n",
       "20,627  100            197               -0.0016      -0.0005      100.0        518.67  643.54  1604.5   1433.58  14.62  21.61  550.86  2388.23  9065.11  1.3    48.04   519.68  2388.22  8136.5   8.5139  0.03    395        2388      100.0        38.3   23.1594  3\n",
       "20,628  100            198               0.0004       0.0          100.0        518.67  643.42  1602.46  1428.18  14.62  21.61  550.94  2388.24  9065.9   1.3    48.09   520.01  2388.24  8141.05  8.5646  0.03    398        2388      100.0        38.44  22.9333  2\n",
       "20,629  100            199               -0.0011      0.0003       100.0        518.67  643.23  1605.26  1426.53  14.62  21.61  550.68  2388.25  9073.72  1.3    48.39   519.67  2388.23  8139.29  8.5389  0.03    395        2388      100.0        38.29  23.064   1\n",
       "20,630  100            200               -0.0032      -0.0005      100.0        518.67  643.85  1600.38  1432.14  14.62  21.61  550.79  2388.26  9061.48  1.3    48.2    519.3   2388.26  8137.33  8.5036  0.03    396        2388      100.0        38.37  23.0522  0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a4d5dfc8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-09-02T09:26:45.436721Z",
     "start_time": "2021-09-02T09:26:45.356092Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>#                                 </th><th>unit_number  </th><th>time_in_cycles  </th><th>setting_1  </th><th>setting_2  </th><th>setting_3  </th><th>T2    </th><th>T24   </th><th>T30    </th><th>T50    </th><th>P2   </th><th>P15  </th><th>P30   </th><th>Nf     </th><th>Nc     </th><th>epr  </th><th>Ps30  </th><th>phi   </th><th>NRf    </th><th>NRc    </th><th>BPR   </th><th>farB  </th><th>htBleed  </th><th>Nf_dmd  </th><th>PCNfR_dmd  </th><th>W31  </th><th>W32    </th><th>RUL  </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td><i style='opacity: 0.6'>0</i>     </td><td>1            </td><td>1               </td><td>0.0023     </td><td>0.0003     </td><td>100.0      </td><td>518.67</td><td>643.02</td><td>1585.29</td><td>1398.21</td><td>14.62</td><td>21.61</td><td>553.9 </td><td>2388.04</td><td>9050.17</td><td>1.3  </td><td>47.2  </td><td>521.72</td><td>2388.03</td><td>8125.55</td><td>8.4052</td><td>0.03  </td><td>392      </td><td>2388    </td><td>100.0      </td><td>38.86</td><td>23.3735</td><td>142  </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>1</i>     </td><td>1            </td><td>2               </td><td>-0.0027    </td><td>-0.0003    </td><td>100.0      </td><td>518.67</td><td>641.71</td><td>1588.45</td><td>1395.42</td><td>14.62</td><td>21.61</td><td>554.85</td><td>2388.01</td><td>9054.42</td><td>1.3  </td><td>47.5  </td><td>522.16</td><td>2388.06</td><td>8139.62</td><td>8.3803</td><td>0.03  </td><td>393      </td><td>2388    </td><td>100.0      </td><td>39.02</td><td>23.3916</td><td>141  </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>2</i>     </td><td>1            </td><td>3               </td><td>0.0003     </td><td>0.0001     </td><td>100.0      </td><td>518.67</td><td>642.46</td><td>1586.94</td><td>1401.34</td><td>14.62</td><td>21.61</td><td>554.11</td><td>2388.05</td><td>9056.96</td><td>1.3  </td><td>47.5  </td><td>521.97</td><td>2388.03</td><td>8130.1 </td><td>8.4441</td><td>0.03  </td><td>393      </td><td>2388    </td><td>100.0      </td><td>39.08</td><td>23.4166</td><td>140  </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>3</i>     </td><td>1            </td><td>4               </td><td>0.0042     </td><td>0.0        </td><td>100.0      </td><td>518.67</td><td>642.44</td><td>1584.12</td><td>1406.42</td><td>14.62</td><td>21.61</td><td>554.07</td><td>2388.03</td><td>9045.29</td><td>1.3  </td><td>47.28 </td><td>521.38</td><td>2388.05</td><td>8132.9 </td><td>8.3917</td><td>0.03  </td><td>391      </td><td>2388    </td><td>100.0      </td><td>39.0 </td><td>23.3737</td><td>139  </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>4</i>     </td><td>1            </td><td>5               </td><td>0.0014     </td><td>0.0        </td><td>100.0      </td><td>518.67</td><td>642.51</td><td>1587.19</td><td>1401.92</td><td>14.62</td><td>21.61</td><td>554.16</td><td>2388.01</td><td>9044.55</td><td>1.3  </td><td>47.31 </td><td>522.15</td><td>2388.03</td><td>8129.54</td><td>8.4031</td><td>0.03  </td><td>390      </td><td>2388    </td><td>100.0      </td><td>38.99</td><td>23.413 </td><td>138  </td></tr>\n",
       "<tr><td>...                               </td><td>...          </td><td>...             </td><td>...        </td><td>...        </td><td>...        </td><td>...   </td><td>...   </td><td>...    </td><td>...    </td><td>...  </td><td>...  </td><td>...   </td><td>...    </td><td>...    </td><td>...  </td><td>...   </td><td>...   </td><td>...    </td><td>...    </td><td>...   </td><td>...   </td><td>...      </td><td>...     </td><td>...        </td><td>...  </td><td>...    </td><td>...  </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>13,091</i></td><td>100          </td><td>194             </td><td>0.0049     </td><td>0.0        </td><td>100.0      </td><td>518.67</td><td>643.24</td><td>1599.45</td><td>1415.79</td><td>14.62</td><td>21.61</td><td>553.41</td><td>2388.02</td><td>9142.37</td><td>1.3  </td><td>47.69 </td><td>520.69</td><td>2388.0 </td><td>8213.28</td><td>8.4715</td><td>0.03  </td><td>394      </td><td>2388    </td><td>100.0      </td><td>38.65</td><td>23.1974</td><td>24   </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>13,092</i></td><td>100          </td><td>195             </td><td>-0.0011    </td><td>-0.0001    </td><td>100.0      </td><td>518.67</td><td>643.22</td><td>1595.69</td><td>1422.05</td><td>14.62</td><td>21.61</td><td>553.22</td><td>2388.05</td><td>9140.68</td><td>1.3  </td><td>47.6  </td><td>521.05</td><td>2388.09</td><td>8210.85</td><td>8.4512</td><td>0.03  </td><td>395      </td><td>2388    </td><td>100.0      </td><td>38.57</td><td>23.2771</td><td>23   </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>13,093</i></td><td>100          </td><td>196             </td><td>-0.0006    </td><td>-0.0003    </td><td>100.0      </td><td>518.67</td><td>643.44</td><td>1593.15</td><td>1406.82</td><td>14.62</td><td>21.61</td><td>553.04</td><td>2388.11</td><td>9146.81</td><td>1.3  </td><td>47.57 </td><td>521.18</td><td>2388.04</td><td>8217.24</td><td>8.4569</td><td>0.03  </td><td>395      </td><td>2388    </td><td>100.0      </td><td>38.62</td><td>23.2051</td><td>22   </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>13,094</i></td><td>100          </td><td>197             </td><td>-0.0038    </td><td>0.0001     </td><td>100.0      </td><td>518.67</td><td>643.26</td><td>1594.99</td><td>1419.36</td><td>14.62</td><td>21.61</td><td>553.37</td><td>2388.07</td><td>9148.85</td><td>1.3  </td><td>47.61 </td><td>521.33</td><td>2388.08</td><td>8220.48</td><td>8.4711</td><td>0.03  </td><td>395      </td><td>2388    </td><td>100.0      </td><td>38.66</td><td>23.2699</td><td>21   </td></tr>\n",
       "<tr><td><i style='opacity: 0.6'>13,095</i></td><td>100          </td><td>198             </td><td>0.0013     </td><td>0.0003     </td><td>100.0      </td><td>518.67</td><td>642.95</td><td>1601.62</td><td>1424.99</td><td>14.62</td><td>21.61</td><td>552.48</td><td>2388.06</td><td>9155.03</td><td>1.3  </td><td>47.8  </td><td>521.07</td><td>2388.05</td><td>8214.64</td><td>8.4903</td><td>0.03  </td><td>396      </td><td>2388    </td><td>100.0      </td><td>38.7 </td><td>23.1855</td><td>20   </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "#       unit_number    time_in_cycles    setting_1    setting_2    setting_3    T2      T24     T30      T50      P2     P15    P30     Nf       Nc       epr    Ps30    phi     NRf      NRc      BPR     farB    htBleed    Nf_dmd    PCNfR_dmd    W31    W32      RUL\n",
       "0       1              1                 0.0023       0.0003       100.0        518.67  643.02  1585.29  1398.21  14.62  21.61  553.9   2388.04  9050.17  1.3    47.2    521.72  2388.03  8125.55  8.4052  0.03    392        2388      100.0        38.86  23.3735  142\n",
       "1       1              2                 -0.0027      -0.0003      100.0        518.67  641.71  1588.45  1395.42  14.62  21.61  554.85  2388.01  9054.42  1.3    47.5    522.16  2388.06  8139.62  8.3803  0.03    393        2388      100.0        39.02  23.3916  141\n",
       "2       1              3                 0.0003       0.0001       100.0        518.67  642.46  1586.94  1401.34  14.62  21.61  554.11  2388.05  9056.96  1.3    47.5    521.97  2388.03  8130.1   8.4441  0.03    393        2388      100.0        39.08  23.4166  140\n",
       "3       1              4                 0.0042       0.0          100.0        518.67  642.44  1584.12  1406.42  14.62  21.61  554.07  2388.03  9045.29  1.3    47.28   521.38  2388.05  8132.9   8.3917  0.03    391        2388      100.0        39.0   23.3737  139\n",
       "4       1              5                 0.0014       0.0          100.0        518.67  642.51  1587.19  1401.92  14.62  21.61  554.16  2388.01  9044.55  1.3    47.31   522.15  2388.03  8129.54  8.4031  0.03    390        2388      100.0        38.99  23.413   138\n",
       "...     ...            ...               ...          ...          ...          ...     ...     ...      ...      ...    ...    ...     ...      ...      ...    ...     ...     ...      ...      ...     ...     ...        ...       ...          ...    ...      ...\n",
       "13,091  100            194               0.0049       0.0          100.0        518.67  643.24  1599.45  1415.79  14.62  21.61  553.41  2388.02  9142.37  1.3    47.69   520.69  2388.0   8213.28  8.4715  0.03    394        2388      100.0        38.65  23.1974  24\n",
       "13,092  100            195               -0.0011      -0.0001      100.0        518.67  643.22  1595.69  1422.05  14.62  21.61  553.22  2388.05  9140.68  1.3    47.6    521.05  2388.09  8210.85  8.4512  0.03    395        2388      100.0        38.57  23.2771  23\n",
       "13,093  100            196               -0.0006      -0.0003      100.0        518.67  643.44  1593.15  1406.82  14.62  21.61  553.04  2388.11  9146.81  1.3    47.57   521.18  2388.04  8217.24  8.4569  0.03    395        2388      100.0        38.62  23.2051  22\n",
       "13,094  100            197               -0.0038      0.0001       100.0        518.67  643.26  1594.99  1419.36  14.62  21.61  553.37  2388.07  9148.85  1.3    47.61   521.33  2388.08  8220.48  8.4711  0.03    395        2388      100.0        38.66  23.2699  21\n",
       "13,095  100            198               0.0013       0.0003       100.0        518.67  642.95  1601.62  1424.99  14.62  21.61  552.48  2388.06  9155.03  1.3    47.8    521.07  2388.05  8214.64  8.4903  0.03    396        2388      100.0        38.7   23.1855  20"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ba258c8",
   "metadata": {},
   "source": [
    "### Export the datasets to HDF5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b5954289",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-09-02T09:26:47.364105Z",
     "start_time": "2021-09-02T09:26:47.252200Z"
    }
   },
   "outputs": [],
   "source": [
    "df_train.export_hdf5('./data/data_train.hdf5')\n",
    "df_test.export_hdf5('./data/data_test.hdf5')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a69d1d8a",
   "metadata": {},
   "source": [
    "The data is ready and now we can start with the modeling process."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}