{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Example usage\n", "\n", "Here we will demonstrate how to effectively use the `pyeda31` package to validate the format of data files and conduct basic exploratory data analysis. The steps include checking data file format, identifying missing values, and generating a summary of the data.\n", "\n", "## Imports" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import csv\n", "import pandas as pd\n", "from pyeda31.check_csv import check_csv\n", "from pyeda31.pymissing_values_summary import missing_values_summary\n", "from pyeda31.data_summary import get_summary_statistics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create an Example CSV File\n", "\n", "To begin, we will create a sample CSV file to demonstrate the functionality of `pyeda`. This file will include some missing values to simulate a typical data scenario. You can find this `sample_data.csv` file [here](https://github.com/UBC-MDS/pyeda/blob/main/docs/sample_data.csv)." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# Define file name\n", "file_name = \"sample_data.csv\"\n", "\n", "# Create data with some empty values\n", "data = [\n", " [\"Name\", \"Age\", \"City\"],\n", " [\"Alice\", \"25\", \"New York\"],\n", " [\"Bob\", \"\", \"Los Angeles\"], # Missing age\n", " [\"Charlie\", \"30\", \"\"], # Missing city\n", " [\"Emily\", \"22\", \"Chicago\"], \n", "]\n", "\n", "# Write data to a CSV file\n", "with open(file_name, mode=\"w\", newline=\"\") as file:\n", " writer = csv.writer(file)\n", " writer.writerows(data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Check if the Data File is in CSV Format\n", "\n", "Before performing any analysis, it is crucial to validate whether the given file has a CSV file extension and whether it can be read by the pandas library. You can verify this using the `check_csv` method. If the file is not in CSV format, an error message will be printed to notify the user." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "if not check_csv(file_name):\n", " raise TypeError(\"The given file either does not have a CSV file extension or cannot be read by the pandas library. Please check the printed error message for more details.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Check if the Data File Has Any Missing Values\n", "\n", "Once the data file format has been verified, the next step is to check whether the dataset contains any missing values using `missing_values_summary`.\n", "\n", "This function will quickly provide a summary including: \n", " • The count of missing values for each column. \n", " • The percentage of missing values for each column." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Age 1 (25.0%)\n", "City 1 (25.0%)\n", "Name: Missing Count (Percentage), dtype: object" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sample_df = pd.read_csv(file_name)\n", "\n", "missing_values_summary(sample_df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Get Data Summary\n", "\n", "Finally, it's time to use the `get_summary_statistics` method to quickly generate the summary statistics of your dataset. You can either specify particular columns to analyze or summarize all columns if no column names are provided. \n", " •\t**For numeric columns**: Key statistics, including mean (rounded to the specified decimal places, default is 3), minimum, maximum, median, mode, and range. \n", "\t•\t**For non-numeric columns**: Frequency-based metrics, including the number of unique values, the most frequent value, and its corresponding count. " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NameAgeCity
num_unique_values4NaN3
most_frequent_valueAliceNaNNew York
frequency_of_most_frequent_value1NaN1
meanNaN25.667NaN
minNaN22.000NaN
maxNaN30.000NaN
medianNaN25.000NaN
modeNaN22.000NaN
rangeNaN8.000NaN
\n", "
" ], "text/plain": [ " Name Age City\n", "num_unique_values 4 NaN 3\n", "most_frequent_value Alice NaN New York\n", "frequency_of_most_frequent_value 1 NaN 1\n", "mean NaN 25.667 NaN\n", "min NaN 22.000 NaN\n", "max NaN 30.000 NaN\n", "median NaN 25.000 NaN\n", "mode NaN 22.000 NaN\n", "range NaN 8.000 NaN" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "get_summary_statistics(sample_df)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AgeCity
mean25.667NaN
min22.000NaN
max30.000NaN
median25.000NaN
mode22.000NaN
range8.000NaN
num_unique_valuesNaN3
most_frequent_valueNaNNew York
frequency_of_most_frequent_valueNaN1
\n", "
" ], "text/plain": [ " Age City\n", "mean 25.667 NaN\n", "min 22.000 NaN\n", "max 30.000 NaN\n", "median 25.000 NaN\n", "mode 22.000 NaN\n", "range 8.000 NaN\n", "num_unique_values NaN 3\n", "most_frequent_value NaN New York\n", "frequency_of_most_frequent_value NaN 1" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "get_summary_statistics(sample_df, col=[\"Age\", \"City\"])" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AgeCity
mean25.7NaN
min22.0NaN
max30.0NaN
median25.0NaN
mode22.0NaN
range8.0NaN
num_unique_valuesNaN3
most_frequent_valueNaNNew York
frequency_of_most_frequent_valueNaN1
\n", "
" ], "text/plain": [ " Age City\n", "mean 25.7 NaN\n", "min 22.0 NaN\n", "max 30.0 NaN\n", "median 25.0 NaN\n", "mode 22.0 NaN\n", "range 8.0 NaN\n", "num_unique_values NaN 3\n", "most_frequent_value NaN New York\n", "frequency_of_most_frequent_value NaN 1" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "get_summary_statistics(sample_df, col=[\"Age\", \"City\"], decimal = 1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion\n", "\n", "The `pyeda31` package offers a user-friendly and efficient solution for validating data files and performing essential exploratory data analysis tasks. From checking file formats and identifying missing values to generating data statistics summaries, this tool simplifies the preprocessing stage, allowing you to focus on extracting deeper insights and making informed decisions. Try it out with your own dataset and experience its ease of use!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.8" } }, "nbformat": 4, "nbformat_minor": 4 }