Why does the Python script not be found when submitting a PyFlink job on YARN?

PyFlink job submission failed on YARN: Analysis of the causes of missing Python scripts and solutions

When submitting a PyFlink job using YARN, you may encounter an error in which the Python script is not found, such as:

<code>2024-05-24 16:38:02,030 info org.apache.flink.client.python.pythondriver [] - pyflink181.zip/pyflink181/bin/python: can't open file 'hdfs://nameservice1/pyflink/wc2.py': [errno 2] no such file or directory</code>

This usually happens when using the following commands:

./flink run-application -t yarn-application \
-dyarn.application.name=flinkcdctestpython\
-dyarn.provided.lib.dirs="hdfs://nameservice1/pyflink/flink-dist-181" \
-pyarch hdfs://nameservice1/pyflink/pyflink181.zip \
-pyclientexec pyflink181.zip/pyflink181/bin/python \
-pyexec pyflink181.zip/pyflink181/bin/python \
-py hdfs://nameservice1/pyflink/wc2.py

In contrast, Java job submission usually does not have problems:

./flink run-application -t yarn-application \
-djobmanager.memory.process.size=1024m \
-dtaskmanager.memory.process.size=1024m \
-dyarn.application.name=flinkcdctest \
-dyarn.provided.lib.dirs="hdfs://nameservice1/pyflink/flink-dist-181" \
hdfs://nameservice1/pyflink/statemachineexample.jar

The Java job was successfully submitted, indicating that the HDFS configuration is correct. The problem is the Python script path or configuration of the PyFlink job.

Troubleshooting and resolution steps

1. Verify the Python script path: Use hdfs dfs -ls hdfs://nameservice1/pyflink/wc2.py command to check whether the script wc2.py exists in the specified HDFS path. If it does not exist, make sure the script is uploaded to the path correctly.

2. Check PyFlink configuration: Double-check -pyarch , -pyclientexec and -pyexec parameters to make sure they point exactly to the PyFlink environment and the Python interpreter. pyflink181.zip must contain all necessary Python libraries and execution environments.

3. Permissions issue: Use hdfs dfs -ls -h hdfs://nameservice1/pyflink/wc2.py to view the HDFS permissions of the script. If the permissions are insufficient, use hdfs dfs -chmod 755 hdfs://nameservice1/pyflink/wc2.py to modify the permissions to ensure that YARN and Flink users have read permissions.

4. Log Analysis: Enable detailed logging of Flink and YARN, analyze error logs for more specific error information and where it occurs.

5. Python environment compatibility: Ensure that the Python version used by PyFlink is consistent with the version used when developing scripts, and avoid incompatible versions that cause the script to be unrecognized or executed.

Through the above steps, systematically troubleshooting the problem and adjusting the configuration according to the actual situation, you can solve the problem that the Python script cannot be found when the PyFlink job is submitted on YARN. If the problem persists, provide more detailed Flink and YARN log information for further analysis.

The above is the detailed content of Why does the Python script not be found when submitting a PyFlink job on YARN?. For more information, please follow other related articles on the PHP Chinese website!