Getting Started¶
echo-dataimporter is a package that facilitates importing data such as time series and events into our PostgreSQL database. It provides a set of data importers classes that standardize the process of importing data from various sources, such as SQL Servers, FTP servers, and more.
The Data Handlers section of this documentation provides detailed information about each data handler available in the package, including how to use them and any necessary configurations.
Concept¶
Since version 1.0.0 the functionality of this package is split into two main classes, listed below.
DataHandler¶
The DataHandler is an abstract class that defines methods that will be used in the same way for the multiple data sources. It's idea is to standardize the way we get features, alarms, etc, from the different instances of the data source type.
As It is an abstract class, the actual definition of the methods is inside of the subclasses such as HuaweiFTPDataHandler.
DataImporter¶
The DataImporter is a class that is used to import data from a data source to the Echo database. It's idea is to standardize the way we import data from the different instances of a data source type.
It receives the objects that will be imported and based on the data source connections it will call the respective Handler classes to import the data and then upload them to the Echo database.
Examples¶
Importing classes¶
To import the package, it's recommended to just import the classes, like below.
from echo_dataimporter import GeDataHandler, DataImporter
Getting alarms¶
To get alarms history and its definitions use the following methods.
-
Sample code
print(handler.alarm_definitions()) print(handler.alarm_history(object_name="CLE-CLE1-01", period=DateTimeRange(datetime(2022, 12, 1), datetime(2022, 12, 2)))) -
Output
SystemType EventNumber SystemTypeId EventShortName EventName EventDescription EventType 442 1x_MBC_MACC_05.05.00.68C 1 1124 EVENT_001 Program start PLC Misc - Program start PLC Fault 102 1x_MBC_MACC_05.05.00.68C 8 1124 EVENT_008 Overvoltage Grid - Overvoltage Fault 103 1x_MBC_MACC_05.05.00.68C 9 1124 EVENT_009 Undervoltage Grid - Undervoltage Fault 118 1x_MBC_MACC_05.05.00.68C 10 1124 EVENT_010 Underfrequency Grid - Underfrequency Fault 117 1x_MBC_MACC_05.05.00.68C 11 1124 EVENT_011 Overfrequency Grid - Overfrequency Fault .. ... ... ... ... ... ... ... 157 WindControlSystem_03.00.08 New Units 43 225 EVENT_043 System initialization Plant System Monitoring - System initialization Warning 241 WindControlSystem_03.00.08 New Units 70 225 EVENT_070 Reactive Power test signal active Plant Volt/VAR/Pf Control - Reactive Power tes... Warning 234 WindControlSystem_03.00.08 New Units 124 225 EVENT_124 Voltage test signal active Plant Volt/VAR/Pf Control - Voltage test signa... Warning 237 WindControlSystem_03.00.08 New Units 126 225 EVENT_126 Power factor test signal active Plant Volt/VAR/Pf Control - Power factor test ... Warning 67 WindControlSystem_03.00.08 New Units 128 225 EVENT_128 Frequency test signal active Plant Frequency Droop - Frequency test signal ... Warning [931 rows x 7 columns] object_name manufacturer_id name description start end alarm_type alarm_responsibility data_source_type_name 0 CLE1-01 182 Start-up No Fault - Start-up 2022-12-01 09:42:24.647 2022-12-01 09:44:41.927 S U sql_ge_scada 1 CLE1-01 183 Load operation No Fault - Load operation 2022-12-01 09:44:41.927 NaT S U sql_ge_scada 0 CLE1-01 181 Spinning position No Fault - Spinning position 2022-12-01 09:41:22.207 2022-12-01 09:42:24.647 S U sql_ge_scada 0 CLE1-01 87 Restart time delay No Fault - Restart time delay 2022-12-01 09:39:17.597 2022-12-01 09:39:44.357 S U sql_ge_scada 2 CLE1-01 87 Restart time delay No Fault - Restart time delay 2022-12-01 09:39:45.757 2022-12-01 09:40:15.717 S U sql_ge_scada 4 CLE1-01 87 Restart time delay No Fault - Restart time delay 2022-12-01 09:40:19.357 2022-12-01 09:40:49.317 S U sql_ge_scada 6 CLE1-01 87 Restart time delay No Fault - Restart time delay 2022-12-01 09:40:52.237 2022-12-01 09:41:22.197 S U sql_ge_scada 1 CLE1-01 6 System OK No Fault - System OK 2022-12-01 09:40:51.557 NaT S U sql_ge_scada 0 CLE1-01 433 Pitch system setup test active Pitch - Pitch system setup test active 2022-12-01 09:39:13.197 2022-12-01 09:40:51.517 S U sql_ge_scada 0 CLE1-01 383 Uncontrol battery test active User - Uncontrol battery test active 2022-12-01 09:37:05.037 2022-12-01 09:40:51.517 S U sql_ge_scada 0 CLE1-01 184 Shut down No Fault - Shut down 2022-12-01 09:38:58.767 2022-12-01 09:39:13.167 S U sql_ge_scada 0 CLE1-01 147 Load shutdown No Fault - Load shutdown 2022-12-01 09:37:05.087 2022-12-01 09:38:58.767 S U sql_ge_scada
Getting features¶
To get feature values and its definitions use the following methods.
-
Sample code
features_df = handler.perfdb.features.definitions.get(object_names=["CLE-CLE1-01"], feature_names=["active_power"]).reset_index( drop=False ) print(handler.feature_definitions()) print(handler.feature_values("CLE-CLE1-01", features_df, DateTimeRange(datetime(2022, 12, 1), datetime(2022, 12, 2)))) -
Output
DataPointId DataPointType Name ShortName Description UnitName UnitDescription 1078 9497 Channel 46KV_Ia 46KV_Ia 46KV_Ia A Ampere 1079 9498 Channel 46KV_Ib 46KV_Ib 46KV_Ib A Ampere 1080 9504 Channel 46KV_Ic 46KV_Ic 46KV_Ic A Ampere 1206 9496 Channel 46KV_PWR_MVAR 46KV_PWR_MVAR 46KV_PWR_MVAR MVAR MVAR 1197 9503 Channel 46KV_PWR_MW 46KV_PWR_MW 46KV_PWR_MW MW Mega Watt ... ... ... ... ... ... ... ... 1012 16502 Statistics Yaw right time total YAW_RIGHT Yaw right time total h hour 842 11865 Statistics last month totalized time WinCtrl appln not ru... LMTH_TTIME_WINCTRL_NOTRUNNING last month totalized time WinCtrl appln not ru... s seconds 876 8956 Statistics o g p kW kilo Watt 521 8957 Statistics raghi n raghi No Unit Type Available 841 11838 Statistics totalized time WindControl application not run... TOTTIME_WINCTRL_NOTRUNNING totalized time WindControl application not run... s seconds [1232 rows x 7 columns] DataPointId active_power TimeStampLocalSystem 2022-12-01 00:00:00 725.074036 2022-12-01 00:10:00 936.473633 2022-12-01 00:20:00 737.717163 2022-12-01 00:30:00 979.945374 2022-12-01 00:40:00 1044.701660 ... ... 2022-12-01 23:10:00 954.873474 2022-12-01 23:20:00 1120.536865 2022-12-01 23:30:00 1454.473877 2022-12-01 23:40:00 1796.445068 2022-12-01 23:50:00 1549.039673 [144 rows x 1 columns]
Importing alarms and features¶
The DataImporter class should be used like presented below.
importer = DataImporter()
importer.alarms_history(DateTimeRange(datetime(2022, 12, 1), datetime(2022, 12, 2)), ["CLE-CLE1-01"])
importer.feature_values(DateTimeRange(datetime(2022, 12, 30), datetime(2023, 1, 5)), ["CLE-CLE1-01"])
This will import the alarms and features from the specified time range and object names into the Echoenergia's Performance PostgreSQL database.