1. Natural Timestamping Using Powerline Electromagnetic Radiation
Yang Li
Advanced Digital Sciences Center, Illinois at Singapore
Rui Tan*
Nanyang Technological University, Singapore
David K.Y. Yau
Singapore University of Technology and Design
Today I will present our work on Natural Timestamping Using Powerline Electromagnetic Radiation. This is a joint work with Rui and David. (Slow)
* Corresponding author

2. Common Notion of Time Critical, fundamental
100 million sensors in buildings by 2019
Roboteam
In the near future, a large number of sensors will be deployed in buildings.
Reaching 100 million in two years.
Modern industries are also bracing the vision of IoT for their production lines.
In all these systems, the Common Notion of Time is a critical and fundamental requirement, which enables us to interpret data and coordinate the actions of the nodes.
Otherwise, the wrong timestamps and clock desynchronization will lead to system choas and even physical damage.
For example, in the right video clips, the robotic arms are highly coordinated, which is driven by their synchronized clocks. If they are desynchronized, they will clash into each other, causing damage and disruption.
Critical, fundamental
Interpret data, coordinate nodes
Wrong timestamps and clock desynchronization
System chaos & damage
[Image credit: renascent3.org, imgflip.com]

3. Existing Approaches Accurate time sources Clock synchronization
GPS, chip-scale atomic clock power hungry, outdoor only, too expensive
Clock synchronization
NTP, PTP, RBS, TPSN, FTSP susceptible to network delay/outage
Clock calibration
Power line EMR, FM Radio, Wi-Fi beacons, light flickering partially achieve common notion of time
There are three catogories of approaches for common notion of time.
The first uses highly accurate time sources, such as GPS and chip-scale atomic clock, however GPS is power consuming, limited to outdoor applications. And atomic clock is still too expensive for wide adoption.
The 2nd is the clock synchronization. There are different clock synchronization protocols for computer networks and sensor networks.
But, They can be easily effected by the network delay and outage.
The last one is Clock calibration. Existing studies have leveraged power line Electromagnetic Radiation, Radio Data System of FM radio, Wi-Fi beacons, and even light flickering to calibrate the clocks of sensor nodes.
However, clock calibration only ensures that the clocks advance at the same speed, so they only partially achieve Common Notion of Time.

4. Natural Timestamp Signal contains unique form during given time
Signal form “encodes” when it is sampled
Electrical network frequency (ENF)
Identical in a region
Fluctuating over time
Pervasive in civil infrastructures
ENF (Hz)
Different from existing approaches, in this work, we study a new concept called Natural timestamp.
A natural timestamp is a signal that gives a unique form during a given time period. In other words, the signal form encodes when it is sampled.
In this work, we try to leverage Electrical Network Frequency as the natural timestamp. This is because of two important properties of ENF. First, at any time, the ENF at different locations in the same power grid are identical; Second, they fluctuate over time. This fluctuation is because of the transient imbalance between power supply and demand.
This figure shows the ENF traces collected by two sensors that are 12 km apart. We can see that their ENF traces are identical over time. And the fluctuation is about 0.1Hz. So, ENF is a promising natural timestamp.
And another advantage of ENF is, it is pervasive in civil infrastructures. So, we could use it for most indoor sensors.
0.1 Hz fluctuation
Time (second)
Two locations 12 km apart Mean error = Hz RMSE = Hz

5. ENF Natural Timestamp Passive timestamping in multimedia forensics
Recordings affected by ENF, seconds accuracy
Active timestamping in sensor systems
Sensor into power socket, sub-ms accuracy
Electromagnetic radiation (EMR) ?
[MM’11]
[ICASSP’14]
Past research has investigated the time information in ENF trace. For example, in multimedia forensics, researchers can extract the ENF trace from multimedia recordings and translate the ENF trace into the time when the recording was made. However, the multimedia recorders are just passively affected by ENF, so these forensic approaches can only achieve seconds accuracy.
In our recent work, we developed a sensor that is plugged into power outlet to actively capture the ENF. And we have achieved sub-millisecond accuracy. However, this approach is limited to the nodes on the power-grid.
In this work, we aim to push the state-of-the art by investigating whether the powerline electromagnetic radiation contains natural timestamps, so that the natural timestamping approach can be used on battery-powered sensors.
[RTSS’16]

6. Challenge EMR contains natural timestamps? How accurate?
0.1 Hz ENF fluctuation vs. 4 Hz EMR fluctuation
EMR contains natural timestamps?
How accurate?
However, the main challenge is extracting ENF trace from the Electromagnetic Radiation (EMR) signal.
In the first picture, the dash line represents the AC voltage signal collected directly from a power outlet. The solid line represents the EMR signal collected by a sensor.
Although we can observe there is a lag between EMR and the voltage signal, EMR oscillates in response to the AC voltage.
However, the shape of EMR signal contains noises.
In the second picture, we show the ENF traces computed from the AC voltage and EMR signals.
We use a simple approach to detect the zero-crossing points and compute the ENF as the inverse of the interval between two neigghbour zero crossing points.
The red curve is the ENF trace computed from the AC voltage.
The blue curse tis the ENF trace computed from the EMR voltage.
From the figure, we can see that the ENF computed from the EMR fluctuates by 4 Hz, which is much more larger than the one computed from AC voltage signal, which is 0.1Hz only.
In this work, the key question we ask is, whether the EMR contains timestamps? If so, how accurate they are.

7. Outline Motivation & background EMR natural timestamping
Implementation and evaluation
Applications
So far, I have introduced the motivation and background of this work, now I will present our approach to EMR timestamping.

8. A Use Case Power grid EMR natural timestamp ENF database node Cyber
(sync’ed to UTC)
Cyber
network
EMR sensor
Powerline
EMR
:15:26.230
Before we go into the technical details, I would like to illustrate the overview of our approach by a use case.
Our natural timestamping system consists of EMR sensors and an ENF database node; EMR sensors can communicate with Database node through a cyber network. And we assume they are within the same power grid. The ENF database node is directly connected to the wall power outlet to capture the ENF trace. Here, we assume it is synchronized to the global time.
The workflow of our approach is as follows.
The EMR sensor captures a segment of EMR signal as natural timestamp, and transmit to the database node.
Then the database node will decode the received natural timestamp into global time and transmit back to the sensor. And the sensor can use the global time to adjust its local clock for example.
Power grid G

9. Hardware High-end platform Low-end platform Raspberry Pi
Audio 44.1kHz
Dedicated EMR sensor
Antenna, signal conditioning
conductor wire
Low-end platform
Z1 mote (MSP430)
Built-in
Antenna: conductor wire
Our research is based on two representative platforms.
The first is a high-end platform that is based on a Raspberry Pi single board computer.
We developed a dedicated EMR sensor consisting of an antenna tuned to the 50 Hz EMR signal and a signal conditioning circuit.
The raspberry pi uses an audio card to sample this EMR sensor.
The second platform is a low-end platform based on the Z1 mote, which has a MSP430 MCU.
We simply use a conductor wire as the EMR antenna and use the built-in ADC to sample the signal.
Later, we will show even this simple platform can effectively sense the natural timestamp.

10. Noise Reduction High-frequency noise Low-frequency noise Amplitude
Sample index
Sample index
Now, we illustrate how we address the noises in the EMR.
The EMR can be affected by high-frequency noises. As illustrated in the figure, because of the high-frequency noise, the EMR signal staggers around the zero line, this may lead to repeated zero-crossing detection.
This issue can be mitigated by using a dead-zone approach.
In this approach, once a zero-crossing point is detected, we will not detect any further zero-crossing points in a following time period, which is the dead zone.
However, this dead-zone approach may miss true zero crossings when the EMR is affected by low-frequency noise. For example, in the right picture, the EMR signal barely crosses the zero line. In this case, the dead-zone approach will miss a true zero crossing point. The repeated or missed zero-crossing detection will negatively affect the ENF extraction.
To deal with these two issues simultaneously, we apply a band-pass filter to remove the high-frequency and low-frequency noises. In these two figures, the blue curves are the filtered signals. Based on the filtered signals, the zero-crossing detection is very robust.
However, the implementation of the band-pass filter on resource-limited platforms is challenging. We will explain this later.
Amplitude
Amplitude
Sample index
Sample index

11. Zero crossing detection Sliding window average
ENF Extraction
raw
EMR
50Hz
EMR
Zero-crossing
points
ENF
trace
Band-pass filter
Zero crossing detection
Sliding window average
ENF (Hz)
So, we apply this signal processing pipeline to extract ENF signal. First, we apply a band-pass filter, then, we detect the zero-crossing points, and use a sliding window approach to compute the average frequency to further improve the robustness against noise.
In this figure, the red curve is the ground truth ENF trace, and the yellow points are the results of the signal processing pipeline. We can see that the result can well track the ground truth. The mean error is just 0.7mHz, and the RMSE is just 44mHz.
Time (second)
Mean error = Hz RMSE = Hz

12. Decoding Natural timestamp → Global Time (UTC)
𝑖 ∗ = argmin 𝑖Є 1,𝑚−𝑛+1 dissimilarity(𝑓,𝑔 𝑖:𝑖+𝑛−1
Decoding error
time
ENF
database trace
w/ UTC timestamps
So far, we have an approach to reliably extract the ENF trace from the noisy EMR signal. Now, I will present our approach to decode an ENF natural timestamp to the global time.
We use an optimization approach to do the decoding, which is given by this equation. But I will skip the detailed explanation and use an example to illustrate.
In this figure, the blue curve represents the ENF signal collected by the database node. This red curve represents the natural timestamp to be decoded. We slide the natural timestamp within the database trace. At each slide position, we compute the dissimilarity between the red curve and the corresponding segment of the blue curve.
Eventually, at this point, we have the lowest dissimilarity. And the time moment of this blue segment is the decoding result.
If the dissimilarity minimum point occurs at a wrong place, it will lead to a decoding error.
And we use this decoding error as the main evaluation metric.

13. Outline Motivation & background EMR natural timestamping
Implementation and evaluation
Applications
So far, I have presented our natural timestamping approach. Now, I will present the implementation and evaluation results.

14. Experiment Methodology
Implementation
Sensor: real-time signal processing
Signal processing throughput
Horner Approach to approximate floating-point computation
Database node: 1 hour selected data to decode
Parameters
Sampling rate
Natural timestamp length
Low-end platform vs. high-end platform
Evaluation
5 sites (1 commercial building, 2 campuses, 2 apartments)
Up to 1 month
We have implemented our signal processing algorithm on both the high-end and low-end platforms. And they run in real-time.
The implementation of real-time signal processing pipeline on the low-end platform is challenging.
For example, the floating point computation needed by the band-pass filter will lead to extremely low signal processing throughput. To address this issue, we use the Horner algorithm to approximate the floating point computation using integer computation. This increases the signal processing throughput to 0.7 kHz.
On the database node, for each natural timestamp, it will select a 1-hour trace from the history to decode.
We evaluate the impact of different parameters on the performance of our timestamping approach, including sampling rate and natural timestamp length, and hardware capability.
Our evaluation was performed at 5 sites including 1 commercial building, 2 campuses, and 2 apartments.
Some evaluation lasted for one month.

15. Natural Timestamp Length
Bar: median
whisker: 95%
Decoding error (ms)
Natural timestamp length
The first experiment evaluates the impact of natural timestamp length on the decoding error.
This figure shows the error bars of the decoding error under different settings of the natural timestamp length. We can see that the decoding errors are generally below 150 milliseconds.
Moreover, a longer timestamp will lead to a lower decoding error.
Because longer timestamp will cause more computation, memory, and communication overhead, this result suggests a trade-off between those overheads and the accuracy of timestamping.
In our later evaluation, we use 5 minutes as our default setting for natural timestamp length.
Longer natural timestamp, smaller decoding error
Compute/RAM/comm. overhead vs. accuracy

16. Sampling Rate High rate Low rate 3.7 kHz Good resolution High overhead
Poor resolution
Low overhead
0.7 kHz
This experiment shows the impact of sampling rate on natural timestamping.
The two figures show the ENF traces captured by a Z1 mote when it samples the EMR signal at 3.7 kHz and 0.7 kHz.
Note that the 0.7kHz is the maximum real-time signal processing throughput of the low-end platform.
The red curve is the ground truth ENF and the blue curve is the extraction result.
We can see with a high sampling rate, we can track the ground truth well but also lead to a high overhead.
When the sampling rate is 0.7kHz, the signal processing can barely track the ground truth ENF. However, the decoding accuracy is still satisfactory, which is about 400 milliseconds.

17. Evaluation in Singapore
2 km
Apartment A
142 ms
56 ms
Campus B
Campus A
Apartment B
220 ms
59 ms
160 ms
37 ms
Commercial building
Database
165 ms
82 ms
146 ms
40 ms
Database
High-end platform
Low-end platform
We have deployed our nodes at 5 sites in Singapore.
The longest distance between the database node and the EMR sensor is about 24 km.
The numbers are the decoding results from two platforms.
This figure shows their median decoding errors.
We can see that the high-end platform gives lower errors. Moreover, we didn’t observe significant impact of distance on the decoding errors.
Numbers: median decoding errors
High-end platform gives lower decoding error
No significant impact of distance on decoding error

18. Evaluation in Home Setup Result Kitchen High-end node 8 kHz
155 ms
Floor
lamp
Laptop
Fridge
338 ms
159 ms
Microwave
Setup
High-end node
8 kHz
5-min timestamp
Bedroom
Washing
machine
262 ms
Bathroom
Induction Heating Cooker
Shower Heater
241 ms
Gas Stove
62 ms
Result
Median decoding errors range from 62 ms to 338 ms
Distance from powerline matters
121 ms
257 ms
214 ms
254 ms
161 ms
156 ms
280 ms
Table fan
WiFi Router
Flat TV
111 ms
In an home deployment, we extensively evaluate the impact of the sensor position on the natural timestamping performance.
We place the sensor at 16 locations in a home, which are marked in this figure. The numbers are the median decoding errors. We can see that the decoding error ranges from 62ms to 338ms.
In particular, we conduct a set of experiments to evaluate the impact of the distance from the powerline on the decoding error. We gradually increase the distance between the node and the TV stand, and we found that the decoding error increases with the distance. This is consistent with our intuition since the EMR signal decays with distance.
120 ms
99 ms
Living Room
High-end Platform
159 ms
Median decoding error

19. Outline Motivation & background EMR natural timestamping
Implementation and evaluation
Applications
Offline time recovery
Run-time clock verification
Lastly, we discuss potential applications of our natural timestamping approach. In the paper, we discussed two applications of offline time recovery, and run-time clock verification.
In the offline time recovery, we can use the natural timestamp to recover the global time of important sensor data during network outage. But because of time limit, I will skip this application and focus on the application of run-time clock verification.

20. Run-Time Clock Verification
Lose synchronization
Power failure, hardware/OS fault, blocked NTP, …
7% sensors in 2 yrs have desynchronization [IPSN’09]
Selected database trace
Natural timestamp
Selected database trace
Natural timestamp
Cyber
network
EMR sensor
ENF database node
(sync’ed to UTC)
Claimed timestamp:
:17:23.950
EMR natural timestamp
Clock offset
Count
Clock offset
Count
Z1:
98.5% accuracy
Real systems always face the practical challenge of losing time synchronization, because various reasons like power failure, hardware fault, OS fault, blocked NTP, et cetera.
For example, in a 2-year deployment of 100 sensors, 7% of the sensors had desynchronization problem.
Our approach can be used to verify the clock of the sensors. Specifically, a sensor transmits a natural timestamp together with its local clock value, to the database node. The local clock value is the time when this natural timestamp is sampled.
Then, the database node can verify the integrity of the clock.
The database node will select an ENF trace from its history based on the claimed clock value.
However, because this claimed clock value may be wrong, we have two possible cases.
First, the natural timestamp is covered by the selected trace. For this case, we can accurately decode the natural timestamp to verify the clock. In the second case, the natural timestamp is not covered by the selected trace and our decoding algorithm will give a wrong result. So, we need to differentiate these two cases.
In our approach, we use a sliding window to generate many sub-natural timestamps from the original natural timestamp and decode each sub-natural timestamp. For the first case, the clock offsets estimated from the decoding results will be concentrated at some value. For the second case, the estimated clock offsets will be spread. Based on this difference, we can differentiate the two cases. On the Z1 mote, the accuracy in differentiating the two cases is 98.5%.
Selected database trace …
Natural timestamp (Original)
Sub-natural timestamps

21. Conclusion Powerline EMR contains natural timestamps Future work
Need signal conditioning
50ms median error on high-end platform
150ms median error on low-end platform
Future work
Regional evaluation
Security of EMR signal (for secure clock sync)
To summarize, in this work, we show that powerline EMR contains natural timestamps. But it needs careful design in signal conditioning. We achieve 50ms median error on high-end platform, and 150ms median error on low-end platform.
In our future work, we plan to conduct evaluation at larger scales, and investigate the security of the EMR signal, to develop a secure clock synchronization approach.
Thanks, now I will take questions.