How To Convert Non-UTC Timestamp Into UNIX Epoch Time In Python

When we ingest API data, the query URI string often takes Unix epoch time (or Unix time) in order to specify the datetime range. The epoch time is the way to represent timestamp as the number of seconds that have elapsed since 1970-01-01 00:00:00 UTC.

When you have an input timestamp string and convert it to the epoch time, it is absolutely critical to understand the time zone of your input string as well as the default server time zone and deal with it correctly. This sounds complicated. But, do not worry. I will show you how to convert a timestamp string into the epoch time in different scenarios.

Examples

Let’s take Melbourne/Australia time zone (AEST) for example. 2018-02-15 00:00:00 in AEST is 2018-02-14 13:00:00 in UTC, which is 1518613200. You can check this with this online epoch time converter.

So, when the input string is 2018-02-15 00:00:00 in AEST, the epoch output should be 1518613200.

For a starter, you need to check the default time zone on your server. Here are the commands.

# Find timezone in Windows Server:
> systeminfo | findstr /C:"Time Zone"

# Find timezone in Linux Server:
> date

Scenario 1: Server time zone is the same as the input string time zone (AEST)

This is the easier scenario. You can use mktime() function from the time module. This takes struct_time in local time and convert to epoch. You need to format the string into the struct_time datatype with strptime() first and use it in mktime().

Assuming the server time zone is AEST, let’s convert 2018-02-15 00:00:00 in AEST to epoch. When you use the input string as 2018-02-15, it assumes the time is 00:00:00. So, both functions below work. You will see the output of 1518613200.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from time import strptime, strftime, mktime, gmtime

def datetime_converter(datetime_string):
    # (1) Convert to datetime format
    target_timestamp = strptime(datetime_string, '%Y-%m-%d %H:%M:%S')

    # (2) mktime creates epoch time from the local time
    mktime_epoch = mktime(target_timestamp)
    print(int(mktime_epoch)) # convert to integer to remove decimal

    # (3) gmtime to convert epoch time into UTC time object
    epoch_to_timestamp = strftime('%Y-%m-%d %H:%M:%S', gmtime(mktime_epoch))
    print(epoch_to_timestamp)

def date_converter(date_string):
    # (1) Convert to date format
    target_timestamp = strptime(date_string, '%Y-%m-%d')

    # (2) mktime creates epoch time from the local time
    mktime_epoch = mktime(target_timestamp)
    print(int(mktime_epoch)) # convert to integer to remove decimal

    # (3) gmtime to convert epoch time into UTC time object
    epoch_to_timestamp = strftime('%Y-%m-%d %H:%M:%S', gmtime(mktime_epoch))
    print(epoch_to_timestamp)

datetime_converter('2018-02-15 00:00:00')
date_converter('2018-02-15')

Scenario 2: Server time zone is UTC and the input string time zone is AEST.

This scenario is slightly more complex. The mktime() uses the local time from the server. Therefore, you first need to convert AEST into UTC before passing it to mktime().

To convert local time to UTC, you can use the pytz module. To begin with, let’s convert local time (2018-02-15 00:00:00) to UTC. You will get the output of 2018-02-14 13:00:00 from the function below.

You first need to define your time zone with pytz.timezone() function. Then, add the time zone to the time_struct object with the localize() function. This can be converted to UTC with astimezone (pytz.utc).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import pytz
from datetime import datetime
import time

def convert_to_utc(timestamp_string, timezone):

    local_tz = pytz.timezone(timezone)
    ts = datetime.strptime(timestamp_string, '%Y-%m-%d %H:%M:%S')
    ts = local_tz.localize(ts)
    ts = str(ts.astimezone(pytz.utc))[0:19]
    print('UTC timestamp is: {}'.format(ts))
    return ts

convert_to_utc('2018-02-15 00:00:00',"Australia/Melbourne")

Ok, once you get the hang of converting your local time to UTC, all you need to do is use mktime() to convert it to epoch. Remember mktime() uses the default local time in the server as UTC. Therefore, passing the time_struct object with UTC time will automatically converts to the right epoch time. In this case, 2018-02-15 00:00:00 in AEST gets converted to 1518613200.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import pytz
from datetime import datetime
import time

def convert_timezone_to_utc_epoch(timestamp_string, timezone):

    local_tz = pytz.timezone(timezone)
    ts = datetime.strptime(timestamp_string, '%Y-%m-%d %H:%M:%S')
    ts = local_tz.localize(ts)
    ts = str(ts.astimezone(pytz.utc))[0:19]
    print('UTC timestamp is: {}'.format(ts))
    epoch = int(time.mktime(time.strptime(ts, '%Y-%m-%d %H:%M:%S')))
    print('Epoch for API call is: {}'.format(epoch))
    return epoch

convert_timezone_to_utc_epoch('2018-02-15',"Australia/Melbourne")

Finally, if your input string is UTC time, you can convert it to epoch from UTC with the function below. When the input string is 2018-02-14 13:00:00 UTC, the function below will give us the same value as print int(time.mktime(time.strptime(‘2018-02-15 00:00:00′,’%Y-%m-%d %H:%M:%S’))).

To convert UTC to epoch, you can use timegm() from calender. Make sure to convert it to the right data type with utctimetuple() prior to using the timegm function.

1
2
3
4
5
6
7
8
9
10
11
import calendar, time
from datetime import datetime

def convert_utc_to_epoch(timestamp_string):
    '''Use this function to convert utc to epoch'''
    timestamp = datetime.strptime(timestamp_string, '%Y-%m-%d %H:%M:%S')
    epoch = int(calendar.timegm(timestamp.utctimetuple()))
    print epoch
    return epoch

convert_utc_to_epoch('2018-02-14 13:00:00')

Finally, here is the quick function to convert epoch to local time.

1
2
3
4
5
6
7
8
from time import strftime, localtime

def convert_epoch_to_local(epoch):
    '''Converting epoch to local time'''
    return strftime('%Y-%m-%d %H:%M:%S', localtime(epoch))

check = convert_epoch_to_local(1522242000)
print(check)

Now, you do not need to fear time zone!

Data Engineering
Downloading All Public GitHub Gist Files

I used to use plug-ins to render code blocks for this blog. Yesterday, I decided to move all the code into GitHub Gist and inject them from there. Using a WordPress plugin to render code blocks can be problematic when update happens. Plugins might not be up to date. It …

Data Engineering
6
Loading Data Frame to Relational Database with R

Once you create a data frame with R, you may need to load it to a relational database for data persistence. You might have a data transformation batch job written in R and want to load database in a certain frequency. Here, I created a function to load data into …

Data Engineering
Exporting LaunchDarkly Flag List into a CSV File with Python

At the moment, LaunchDarkly does not have functionality to export a list of flags as csv or excel file. This can change very near future (it may already have the functionality by the time you are reading this post). The workaround is to use API to ingest the data. Here …