This notebook contains an examples of creating a UDF in Python and registering it for use in Spark SQL.
def squared(s): return s * s spark.udf.register("squaredWithPython", squared)
Optionally, you can also explicitly set the return type of your UDF.
from pyspark.sql.types import LongType def squared_typed(s): return s * s spark.udf.register("squaredWithPython", squared, LongType())
%sql select id, squaredWithPython(id) as id_squared from test