An earlier post shows how to run Pyspark (Spark 2.1.0) in Eclipse (ver 2024-06 (4.32)) using the PyDev (ver 12.1) plugin. The OS is Ubuntu-20.04 with Java-8, & an older version of Python3.5 compatible with PySpark (2.1.0).
While the Pyspark code runs fine within Eclipse, when trying to Debug an error is thrown:
Pydev: Unexpected error setting up the debugger: Socket closed".
This is due to a higher Python requirement (>3.6) for pydevd debugger module within PyDev. Details from the PyDev installations page clearly state that Python3.5 is compatible only with PyDev9.3.0. So it's back to square one.
Install/ replace Pydev 12.1 with PyDev 9.3 in Eclipse
- Uninstall Pydev 12.1 (Help > About > Installation details > Installed software > Uninstall PyDev plugin)
- Also manually remove all Pydev folders from eclipse/plugins folder (com.python.pydev.* & org.python.pydev.*)
- Download & Install PyDev 9.3 from Download location as per instructions
- Unzip to eclipse/dropins folder
- Restart eclipse & check (Help > About > Installation details > Installed software)
Test debugging Pyspark
Refer to the steps to Run Pyspark on PyDev in Eclipse, & ensure the PyDev Interpreter is python3.5, PYSPARK_PYTHON variable and PYTHONPATH are correctly setup.
Finally, right click on network_wordcount.py > Debug as > Python run
(Set up Debug Configurations > Arguments & provide program arguments, e.g. "localhost 9999", & any breakpoints in the python code to test).
No comments:
Post a Comment