Backend Development
Python Tutorial
Develop a system that monitors a log file located on a remote server, similar to the Unix command tail -f.
Develop a system that monitors a log file located on a remote server, similar to the Unix command tail -f.
The Objective of this problem is:
The Objective is to develop a system that monitors a log file located on a remote server, similar to the Unix command tail -f. The log file is being continuously appended with new data. This system should consist of:
A server application that tracks the ongoing changes to a specified log file situated on the same server. This application should be capable of transmitting the newly added data in real-time to clients.
A web-based client interface, reachable via a URL (e.g., http://localhost/log), designed to display log file updates dynamically as they occur, without requiring the user to reload the page. Initially, upon visiting the page, users should see the most recent 10 lines from the log file.
Also Handled the following scenarios:The server must actively push updates to the clients to ensure minimal delay, achieving as close to real-time updates as possible.
Given that the log file might be very large (potentially several gigabytes), you'll need to develop a strategy for efficiently fetching the last 10 lines without processing the entire file.
The server should transmit only new additions to the file to the clients, rather than resending the file in its entirety.
It's essential that the server supports concurrent connections from multiple clients without performance degradation.
The client's web page should load promptly without staying in a loading state after the initial request, and it should not require reloading to display new updates.
I have created a Flask application with a simple UI that displays the last 10 messages.
i have used the flask-socketio to form a connection also used the some basic concepts of handling files like fileObj.seek(), fileObj.tell() etc.
from flask import Flask, render_template
from flask_socketio import SocketIO, emit
from threading import Lock
app = Flask(__name__)
socketio = SocketIO(app)
thread = None
thread_lock = Lock()
LOG_FILE_PATH = "./static/client.txt"
last_position = 0
position_lock = Lock()
@app.route('/')
def index():
return render_template('index.html')
@socketio.on('connect')
def test_connect():
global thread
with thread_lock:
if thread is None:
print("started execution in background!")
thread = socketio.start_background_task(target=monitor_log_file)
def monitor_log_file():
global last_position
while True:
try:
with open(LOG_FILE_PATH, 'rb') as f:
f.seek(0, 2)
file_size = f.tell()
if last_position != file_size:
buffer_size = 1024
if file_size < buffer_size:
buffer_size = file_size
f.seek(-buffer_size, 2)
lines = f.readlines()
last_lines = lines[-10:]
content = b'\n'.join(last_lines).decode('utf-8')
socketio.sleep(1) # Add a small delay to prevent high CPU usage
socketio.emit('log_updates', {'content': content})
print("Emitted new Lines to Client!")
last_position = file_size
else:
pass
except FileNotFoundError:
print(f"Error: {LOG_FILE_PATH} not found.")
except Exception as e:
print(f"Error while reading the file: {e}")
if __name__ == '__main__':
socketio.run(app, debug=True, log_output=True, use_reloader=False)
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Basics</title>
<script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.5.4/socket.io.js"></script>
</head>
<body>
<h1>User Updated Files Display it over here:</h1>
<div id="output"></div>
<script>
var socket = io("http://127.0.0.1:5000");
socket.on('connect', function() {
console.log('Connected to the server');
});
socket.on('disconnect', function() {
console.log('Client disconnected');
});
socket.on('log_updates', function(data) {
console.log("data", data);
var div = document.getElementById('output');
var lines = data.content.split('\n');
div.innerHTML = '';
lines.forEach(function(line) {
var p = document.createElement('p');
p.textContent = line;
div.appendChild(p);
});
});
</script>
</body>
</html>
Also create a client.log file under static folder in flask application.
Please feel free to correct me if I did something wrong. Comment below with any corrections!
The above is the detailed content of Develop a system that monitors a log file located on a remote server, similar to the Unix command tail -f.. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
How to solve the permissions problem encountered when viewing Python version in Linux terminal?
Apr 01, 2025 pm 05:09 PM
Solution to permission issues when viewing Python version in Linux terminal When you try to view Python version in Linux terminal, enter python...
How to avoid being detected by the browser when using Fiddler Everywhere for man-in-the-middle reading?
Apr 02, 2025 am 07:15 AM
How to avoid being detected when using FiddlerEverywhere for man-in-the-middle readings When you use FiddlerEverywhere...
How to teach computer novice programming basics in project and problem-driven methods within 10 hours?
Apr 02, 2025 am 07:18 AM
How to teach computer novice programming basics within 10 hours? If you only have 10 hours to teach computer novice some programming knowledge, what would you choose to teach...
How to efficiently copy the entire column of one DataFrame into another DataFrame with different structures in Python?
Apr 01, 2025 pm 11:15 PM
When using Python's pandas library, how to copy whole columns between two DataFrames with different structures is a common problem. Suppose we have two Dats...
How does Uvicorn continuously listen for HTTP requests without serving_forever()?
Apr 01, 2025 pm 10:51 PM
How does Uvicorn continuously listen for HTTP requests? Uvicorn is a lightweight web server based on ASGI. One of its core functions is to listen for HTTP requests and proceed...
How to handle comma-separated list query parameters in FastAPI?
Apr 02, 2025 am 06:51 AM
Fastapi ...
How to solve permission issues when using python --version command in Linux terminal?
Apr 02, 2025 am 06:36 AM
Using python in Linux terminal...
How to get news data bypassing Investing.com's anti-crawler mechanism?
Apr 02, 2025 am 07:03 AM
Understanding the anti-crawling strategy of Investing.com Many people often try to crawl news data from Investing.com (https://cn.investing.com/news/latest-news)...
