How to Convert an int96 Parquet Timestamp to Go's time.Time?

Casting int96 Timestamp from Parquet to Go

Introduction

When working with timestamps stored in parquet files, it's common to encounter data represented as an int96 datatype. This 12-byte format consists of a timestamp and date portion. Understanding how to cast this int96 timestamp to a timestamp in Go is crucial for data manipulation and analysis.

Decoding int96 Timestamp

The key to decoding an int96 timestamp lies in recognizing its structure. The first eight bytes represent the timestamp as nanoseconds, but are stored in reverse byte order.

time_nano = ReverseBytes((timestamp[0:8]))

The remaining four bytes represent the date as a Julian day number.

julian_day = ReverseBytes((timestamp[8:12]))

Example Calculation

Consider the following int96 timestamp: [128, 76, 69, 116, 64, 7, 0, 0, 48, 131, 37, 0].

• Time (nanoseconds): Reversing the first eight bytes gives 0x60FD4B3229000059682500, which corresponds to 45296 seconds (i.e., 12 hours, 34 minutes, 56 seconds).
• Date (Julian day number): Reversing the last four bytes gives 0x00256859, corresponding to Julian day number 2451545, which equals 2000-01-01.

Golang Code Implementation

Here's a Go code snippet that demonstrates the decoding process:

package main

import (
    "fmt"
    "math"
)

func main() {
    // Sample int96 timestamp
    timestamp := []byte{128, 76, 69, 116, 64, 7, 0, 0, 48, 131, 37, 0}

    // Reverse the first 8 bytes for time (nanoseconds)
    timeNano := reverseBytes(timestamp[0:8])

    // Reverse the last 4 bytes for date (Julian day number)
    julianDay := reverseBytes(timestamp[8:12])

    // Convert timestamp nanoseconds to seconds
    seconds := float64(timeNano) / math.Pow(10, 9)

    // Calculate the timestamp
    t := time.Unix(int64(seconds), 0)

    // Create a new time object with date set from Julian day number
    t = t.AddDate(int(julianDay/365.25-2440587.5), 0, int(julianDay%365.25))

    fmt.Println(t)
}

func reverseBytes(bytes []byte) uint64 {
    var result uint64
    for _, v := range bytes {
        result = result << 8 | uint64(v)
    }
    return result
}

The above is the detailed content of How to Convert an int96 Parquet Timestamp to Go's time.Time?. For more information, please follow other related articles on the PHP Chinese website!