{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "011002f2",
   "metadata": {},
   "source": [
    "# Auto-complete & easier refactoring using schema attributes \n",
    "Schemas allow us to replaces the numerous strings throughout the code by schema attributes. Consider the following example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6379b38b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typedspark import Column, DataSet, Schema\n",
    "from pyspark.sql.types import LongType, StringType\n",
    "from pyspark.sql.functions import col\n",
    "\n",
    "\n",
    "class Person(Schema):\n",
    "    id: Column[LongType]\n",
    "    name: Column[StringType]\n",
    "    age: Column[LongType]\n",
    "\n",
    "\n",
    "def birthday(df: DataSet[Person]) -> DataSet[Person]:\n",
    "    return DataSet[Person](\n",
    "        df.withColumn(\"age\", col(\"age\") + 1),\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a03a164a",
   "metadata": {},
   "source": [
    "We can replace this with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1e6fa606",
   "metadata": {},
   "outputs": [],
   "source": [
    "def birthday(df: DataSet[Person]) -> DataSet[Person]:\n",
    "    return DataSet[Person](\n",
    "        df.withColumn(Person.age.str, Person.age + 1),\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4232e8ce",
   "metadata": {},
   "source": [
    "Which allows:\n",
    "\n",
    "* Autocomplete of column names during coding\n",
    "* Easy refactoring of column names\n",
    "\n",
    "Note that we have two options when using schema attributes:\n",
    "\n",
    "* `Person.age`, which is similar to a Spark `Column` object (i.e. `col(\"age\")`)\n",
    "* `Person.age.str`, which is just the column name (i.e. `\"age\"`)\n",
    "\n",
    "It is usually fairly obvious which one to use. For instance, in the above example, `withColumn()` expects a string as the first argument and a `Column` object as the second argument."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a56cffb",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "typedspark",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}