Table of Contents
Conclusion
Home Backend Development Python Tutorial Get and store time series data with Python

Get and store time series data with Python

Apr 11, 2023 pm 07:30 PM
python time series data

Get and store time series data with Python

##Translator| Bugatti

Reviewer| Sun Shujuan

This tutorial will introduce how to Use Python to get time series data from the OpenWeatherMap API and convert it into a Pandas DataFrame. Next, we will use the InfluxDB Python Client to write this data to the time series data platform InfluxDB.

We will convert the JSON response from the API call into a Pandas DataFrame as this is the easiest way to write data to InfluxDB. Since InfluxDB is a purpose-built database, our writes to InfluxDB are designed to meet the high requirements in terms of ingestion of time series data.

Requirements

This tutorial is completed on a macOS system that has Python 3 installed via Homebrew. It is recommended to install additional tools such as virtualenv, pyenv or conda-env to simplify the installation of Python and Client. The full requirements are here:

1

2

3

4

txt

influxdb-client=1.30.0

pandas=1.4.3

requests>=2.27.1

Copy after login

This tutorial also assumes that you have already created a Free Tier InfluxDB cloud account or are using InfluxDB OSS, and that you have also:

    Created the bucket. You can think of buckets as the highest level of data organization in a database or InfluxDB.
  • Token created.
Finally, this tutorial requires that you have already created an account with OpenWeatherMap and created a token.

Request weather data

First, we need to request data. We will use the requests library to return hourly weather data from a specified longitude and latitude via the OpenWeatherMap API.

1

2

3

4

5

# Get time series data from OpenWeatherMap API

params = {'lat':openWeatherMap_lat, 'lon':openWeatherMap_lon, 'exclude':

"minutely,daily", 'appid':openWeatherMap_token}

r = requests.get(openWeather_url, params = params).json()

hourly = r['hourly']

Copy after login

Convert data into Pandas DataFrame

Next, convert the JSON data into Pandas DataFrame. We also convert the timestamp from a second-precision Unix timestamp to a datetime object. This conversion is done because the InfluxDB write method requires the timestamp to be in datetime object format. Next, we will use this method to write data to InfluxDB. We also removed columns that we didn't want written to InfluxDB.

1

2

3

4

5

6

7

python

# Convert data to Pandas DataFrame and convert timestamp to datetime

object

df = pd.json_normalize(hourly)

df = df.drop(columns=['weather', 'pop'])

df['dt'] = pd.to_datetime(df['dt'], unit='s')

print(df.head)

Copy after login

Writing Pandas DataFrame to InfluxDB

Now create an instance of the InfluxDB Python client library and write the DataFrame to InfluxDB. We specified the measurement name. Measurements contain data in buckets. You can think of it as the second-highest level structure in InfluxDB's data organization after buckets.

You can also use the data_frame__tag_columns parameter to specify which columns are converted to tags.

Since we did not specify any columns as labels, all of our columns will be converted into fields in InfluxDB. Tags are used to write metadata about your time series data, which can be used to query subsets of the data more efficiently. Fields are where you store the actual time series data in InfluxDB. This document (https://docs.influxdata.com/influxdb/cloud/reference/key-concepts/?utm_source=vendor&utm_medium=referral&utm_campaign=2022-07_spnsr-ctn_obtaining-storing-ts-pything_tns) goes into more detail about These data concepts.

1

2

3

4

5

6

7

on

# Write data to InfluxDB

with InfluxDBClient(url=url, token=token, org=org) as client:

df = df

client.write_api(write_options=SYNCHRONOUS).write(bucket=bucket,record=df,

data_frame_measurement_name="weather",

data_frame_timestamp_column="dt")

Copy after login

Full script

To review, you might as well take a look at the complete script. We take the following steps:

1. Import the library.

2. Collect the following:

    InfluxDB Bucket
  • InfluxDB Organization
  • InfluxDB Token
  • InfluxDB URL
  • OpenWeatherMap URL
  • OpenWeatherMap Token
3. Create the request.

4. Convert JSON response into Pandas DataFrame.

5. Delete any columns that you do not want to write to InfluxDB.

6. Convert timestamp column from Unix time to Pandas datetime object.

7. Create an instance for the InfluxDB Python Client library.

8. Write a DataFrame and specify the measurement name and timestamp column.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

python

import requests

import influxdb_client

import pandas as pd

from influxdb_client import InfluxDBClient

from influxdb_client.client.write_api import SYNCHRONOUS

bucket = "OpenWeather"

org = "" # or email you used to create your Free Tier

InfluxDB Cloud account

token = "

url = "" # for example,

https://us-west-2-1.aws.cloud2.influxdata.com/

openWeatherMap_token = ""

openWeatherMap_lat = "33.44"

openWeatherMap_lon = "-94.04"

openWeather_url = "https://api.openweathermap.org/data/2.5/onecall"

# Get time series data from OpenWeatherMap API

params = {'lat':openWeatherMap_lat, 'lon':openWeatherMap_lon, 'exclude':

"minutely,daily", 'appid':openWeatherMap_token}

r = requests.get(openWeather_url, params = params).json()

hourly = r['hourly']

# Convert data to Pandas DataFrame and convert timestamp to datetime

object

df = pd.json_normalize(hourly)

df = df.drop(columns=['weather', 'pop'])

df['dt'] = pd.to_datetime(df['dt'], unit='s')

print(df.head)

# Write data to InfluxDB

with InfluxDBClient(url=url, token=token, org=org) as client:

df = df

client.write_api(write_options=SYNCHRONOUS).write(bucket=bucket,record=df,

data_frame_measurement_name="weather",

data_frame_timestamp_column="dt")

Copy after login

Query data

Now that we have written the data to InfluxDB, we can use the InfluxDB UI to query the data. Navigate to Data Explorer (from the left navigation bar). Using Query Builder, select the data you want to visualize and the range you want to visualize, and click Submit.

Get and store time series data with Python

Figure 1. Default materialized view of weather data. InfluxDB automatically aggregates time series data so new users don't accidentally query too much data and cause timeouts

Pro Tip: When you query data using the query builder, InfluxDB automatically downsamples the data. To query raw data, navigate to the Script Editor to view the underlying Flux query. Flux is a native query and scripting language for InfluxDB that can be used to analyze and create predictions using your time series data. Use the aggregateWindow() function to uncomment or delete rows to see the original data.

Get and store time series data with Python

Figure 2. Navigate to the Script Editor and uncomment or delete the aggregateWindow() function to view the raw weather data

Conclusion

Hopefully this article helped you get the most out of InfluxDB Python Client library, obtains time series data and stores it in InfluxDB. If you want to learn more about using the Python Client library to query data from InfluxDB, I recommend you take a look at this article (https://thenewstack.io/getting-started-with-python-and-influxdb/). It's also worth mentioning that you can use Flux to get data from the OpenWeatherMap API and store it into InfluxDB. If you use InfluxDB Cloud, this means that the Flux script will be hosted and executed periodically, so you can get a reliable stream of weather data fed into the instance. To learn more about how to use Flux to obtain weather data on a user-defined schedule, please read this article (https://www.influxdata.com/blog/tldr-influxdb-tech-tips-handling-json-objects-mapping- arrays/?utm_source=vendor&utm_medium=referral&utm_campaign=2022-07_spnsr-ctn_obtaining-storing-ts-pything_tns).

The above is the detailed content of Get and store time series data with Python. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Choosing Between PHP and Python: A Guide Choosing Between PHP and Python: A Guide Apr 18, 2025 am 12:24 AM

PHP is suitable for web development and rapid prototyping, and Python is suitable for data science and machine learning. 1.PHP is used for dynamic web development, with simple syntax and suitable for rapid development. 2. Python has concise syntax, is suitable for multiple fields, and has a strong library ecosystem.

PHP and Python: Different Paradigms Explained PHP and Python: Different Paradigms Explained Apr 18, 2025 am 12:26 AM

PHP is mainly procedural programming, but also supports object-oriented programming (OOP); Python supports a variety of paradigms, including OOP, functional and procedural programming. PHP is suitable for web development, and Python is suitable for a variety of applications such as data analysis and machine learning.

Can visual studio code be used in python Can visual studio code be used in python Apr 15, 2025 pm 08:18 PM

VS Code can be used to write Python and provides many features that make it an ideal tool for developing Python applications. It allows users to: install Python extensions to get functions such as code completion, syntax highlighting, and debugging. Use the debugger to track code step by step, find and fix errors. Integrate Git for version control. Use code formatting tools to maintain code consistency. Use the Linting tool to spot potential problems ahead of time.

Can vs code run in Windows 8 Can vs code run in Windows 8 Apr 15, 2025 pm 07:24 PM

VS Code can run on Windows 8, but the experience may not be great. First make sure the system has been updated to the latest patch, then download the VS Code installation package that matches the system architecture and install it as prompted. After installation, be aware that some extensions may be incompatible with Windows 8 and need to look for alternative extensions or use newer Windows systems in a virtual machine. Install the necessary extensions to check whether they work properly. Although VS Code is feasible on Windows 8, it is recommended to upgrade to a newer Windows system for a better development experience and security.

Is the vscode extension malicious? Is the vscode extension malicious? Apr 15, 2025 pm 07:57 PM

VS Code extensions pose malicious risks, such as hiding malicious code, exploiting vulnerabilities, and masturbating as legitimate extensions. Methods to identify malicious extensions include: checking publishers, reading comments, checking code, and installing with caution. Security measures also include: security awareness, good habits, regular updates and antivirus software.

How to run programs in terminal vscode How to run programs in terminal vscode Apr 15, 2025 pm 06:42 PM

In VS Code, you can run the program in the terminal through the following steps: Prepare the code and open the integrated terminal to ensure that the code directory is consistent with the terminal working directory. Select the run command according to the programming language (such as Python's python your_file_name.py) to check whether it runs successfully and resolve errors. Use the debugger to improve debugging efficiency.

PHP and Python: A Deep Dive into Their History PHP and Python: A Deep Dive into Their History Apr 18, 2025 am 12:25 AM

PHP originated in 1994 and was developed by RasmusLerdorf. It was originally used to track website visitors and gradually evolved into a server-side scripting language and was widely used in web development. Python was developed by Guidovan Rossum in the late 1980s and was first released in 1991. It emphasizes code readability and simplicity, and is suitable for scientific computing, data analysis and other fields.

Python vs. JavaScript: The Learning Curve and Ease of Use Python vs. JavaScript: The Learning Curve and Ease of Use Apr 16, 2025 am 12:12 AM

Python is more suitable for beginners, with a smooth learning curve and concise syntax; JavaScript is suitable for front-end development, with a steep learning curve and flexible syntax. 1. Python syntax is intuitive and suitable for data science and back-end development. 2. JavaScript is flexible and widely used in front-end and server-side programming.

See all articles