The Environmental Monitoring and Analysis (EMA) Project is a Industry Sponsored Student Project (ISSP) in partnership with the BCIT Centre for Internet of Things (IoT) and the YVR Airport Innovation Hub.
The Project aims to develop and deploy cloud infrastructure for IoT network devices that monitor environmental data in the canals around YVR Airport in Vancouver, British Columbia.
The technologies for this project are hosted by Amazon Web Services (AWS) cloud applications.
1. Project Overview
The Environmental Monitoring and Analysis (EMA) Project is a Industry Sponsored Student Project (ISSP) in partnership with the BCIT Centre for Internet of Things (IoT) and the YVR Airport Innovation Hub.
The Project aims to develop and deploy cloud infrastructure for IoT network devices that monitor environmental data in the canals around YVR Airport in Vancouver, British Columbia.
The technologies for this project are hosted by Amazon Web Services (AWS) cloud applications. Application services include Iot Core, Kinesis Data stream and analytics, AppSync, and Timestream database.
Sensor devices are network connected to an edge computing device that relays all collected data to the cloud for analytics and processing, before being returned to a dashboard monitoring service.
Project Initial Architecture:
The Cloud Infrastructure Diagram
Demo
AWS Ecosystem:
IoT Core: Establishes a connection with data sensors to receive data and route them over to Amazon Web Services’s (AWS) cloud infrastructure.
IoT Things: An IoT thing is a representation and record of your physical device in the cloud. A physical device needs a thing record in order to work with AWS IoT.
We currently have two IoT things in our architecture:
- YVR_water_sensor1 (Javascript programed sensor)
- YVR_water_sensor2 (Python programed sensor)
Both simulator sensors are stored as an AWS EC2 instance AMI
IoT Core MQTT Protocol: A protocol used to establish connections with the end device through AWS Greengrass.
Both sensor simulators are posting data to an AWS topic “water_sensor1”
IoT Rules: Rules give our devices the ability to interact with AWS services. Rules are analyzed and actions are performed based on the MQTT topic stream.
We currently have three IoT rules in our architecture:
- Sensor_fulldata_timestream is responsible for writing raw data to the timestream database
- sensor_filltered_timestream is responsible for writing filtered data to the timestream database
- YVR_water_sensor1 is responsible for sending data to AWS Kinesis data stream.
Kinesis Data: Reroutes and modifies data using SQL statements.
Kinesis Data Stream: Pipes the data from AWS IoT Core into other Kinesis services
We have one main data stream “YVR_water_sensor_main_stream”
Kinesis Data Firehose: Routes the data received into Amazon S3 Buckets
We have two Kinesis firehoses:
- YVR_water_sensor_raw_data_to_s3 : sending the raw data from Kinesis Data Stream to S3 bucket
- Filtered_Data_Stream: sending the filtered data from Kinesis Data Analytics result
Kinesis Data Analytics: Applies SQL statements to modify or filter data.
We have one SQL application within Kinesis Analytics:
- The “watersensor_filter” application uses SQL statements to filter the incoming data before sending the data to kinesis firehose
- The SQL statements can be modified to fit any specific data filtration scenarios
Lambda Functions: Automatically executes code and pipes results to S3 bucket.
- Lambda Function (datatransform): Converts the data received from the Kinesis data hose into JSON String.
- Lambda Function (convertjson): Formats the JSON string to be more readable as a JSON file.
S3 Buckets: A storage used to store files containing data but can also store images.
Watersensor-raw-data: store all raw data ( JSON string) from sensors
Watersensor-analyzed-data: store filtered data from Kinesis data Analytics
Watersensor-raw-data.json: Readable JSON format data
Yvr-water-sensor-error-bucket: store all error logs generated from the whole architecture
TimestreamDB: A database located on AWS. Data can be queried using SQL statements.
All data will be stored in this database YVR_water_sensor
There are two tables in the database:
Water_sensor_fulldata: receive data from rule sensor_fulldata_timestream
Water_sensor_filltered: receive data from rule sensor_filltered_timestream
Other services:
Grafana: A service outside of AWS that automatically creates a dashboard with the data provided.