{
"info": {
"author": "Keshav Shetty",
"author_email": "keshavshetty@gmail.com",
"bugtrack_url": null,
"classifiers": [
"License :: OSI Approved :: MIT License",
"Programming Language :: Python",
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 3"
],
"description": "Chart + Util = Chartil (Click to expand)
\n\n\n# Chart + Util = Chartil\nData visualization: Simple, Single unified API for plotting and charting\n\nDuring EDA/data preparation we use few common and fixed set of chart types to analyse the relation among various features. \nFew are simple charts like univariate and some are complex 3D or even multiple features>3.\n\nThis api is simple, single api to plot various type of relations which will hide all the technical/code details from Data Science task and approch.\nThis overcomes the difficulties of maintaining several api or libraries and avoid repeated codes. \n\nUsing this approach we just need one api (Rest all decided by library)\n\n\tfrom KUtils.eda import chartil\n\n chartil.plot(dataframe, [list of columns]) or\n chartil.plot(dataframe, [list of columns], {optional_settings})\n\n\nDemo code:\n\n# Load UCI Dataset. Download [From here](https://archive.ics.uci.edu/ml/datasets/Heart+Disease/)\n\theart_disease_df = pd.read_csv('../input/uci/heart.csv')\n\n\n# Quick data preparation\n\tcolumn_to_convert_to_categorical = ['target', 'cp', 'fbs', 'exang', 'restecg', 'slope', 'ca', 'thal']\n\tfor col in column_to_convert_to_categorical:\n\t\theart_disease_df[col] = heart_disease_df[col].astype('category')\n\n\theart_disease_df['age_bin'] = pd.cut(heart_disease_df['age'], [0, 32, 40, 50, 60, 70, 100], labels=['<32', '33-40','41-50','51-60','61-70', '71+']) \n\n\theart_disease_df['sex'] = heart_disease_df['sex'].map({1:'Male', 0:'Female'})\n\n\theart_disease_df.info()\n\n# Heatmap\n\tchartil.plot(heart_disease_df, heart_disease_df.columns) # Send all column names \n\n\n\tchartil.plot(heart_disease_df, heart_disease_df.columns, optional_settings={'include_categorical':True} ) \n\n\n\tchartil.plot(heart_disease_df, heart_disease_df.columns, optional_settings={'include_categorical':True, 'sort_by_column':'trestbps'} ) \n\n\n\t# Force to plot heatmap when you have fewer columns, otherwise tool will decide as different chart\n\tchartil.plot(heart_disease_df, ['chol', 'thalach', 'trestbps'], chart_type='heatmap') \n\n\n# Uni-categorical \n\tchartil.plot(heart_disease_df, ['target']) # Barchart as count plot \n\n\n# Uni-Continuous\n\tchartil.plot(heart_disease_df, ['age'])\n\n\n\tchartil.plot(heart_disease_df, ['age'], chart_type='barchart') # Force barchart on cntinuous by auto creating 10 equal bins \n\n\n\tchartil.plot(heart_disease_df, ['age'], chart_type='barchart', optional_settings={'no_of_bins':5}) # Create custom number of bins \n\n\n\tchartil.plot(heart_disease_df, ['age'], chart_type='distplot') \n\n\n# Uni-categorical with optional_settings\n\tchartil.plot(heart_disease_df, ['age_bin']) # Barchart as count plot\n\n\n\tchartil.plot(heart_disease_df, ['age_bin'], optional_settings={'sort_by_value':True})\n\n\n\tchartil.plot(heart_disease_df, ['age_bin'], optional_settings={'sort_by_value':True, 'limit_bars_count_to':5})\n\n\n# Bi Category vs Category (& Univariate Segmented)\n\tchartil.plot(heart_disease_df, ['sex', 'target'])\n\n\n\tchartil.plot(heart_disease_df, ['sex', 'target'], chart_type='crosstab')\n\n\n\tchartil.plot(heart_disease_df, ['sex', 'target'], chart_type='stacked_barchart')\n\n\n# Bi Continuous vs Continuous\n\tchartil.plot(heart_disease_df, ['chol', 'thalach']) # Scatter plot\n\n\n# Bi Continuous vs Category\n\tchartil.plot(heart_disease_df, ['thalach', 'sex']) # Grouped box plot (Segmented univariate)\n\n\n\tchartil.plot(heart_disease_df, ['thalach', 'sex'], chart_type='distplot') # Distplot\n\n\n# Multi 3 Continuous\n\tchartil.plot(heart_disease_df, ['chol', 'thalach', 'trestbps']) # Colored 3D scatter plot\n\n\n# Multi 3 Categorical\n\tchartil.plot(heart_disease_df, ['sex', 'age_bin', 'target']) # Paired barchart\n\n\n# Multi 2 Continuous, 1 Category\n\tchartil.plot(heart_disease_df, ['chol', 'thalach', 'target']) # Scatter plot with colored groups \n\n\n# Multi 1 Continuous, 2 Category\n\tchartil.plot(heart_disease_df, ['thalach', 'sex', 'target']) # Grouped boxplot\n\n\n\tchartil.plot(heart_disease_df, ['thalach', 'sex', 'target'], chart_type='violinplot') # Grouped violin plot\n\n\n# Multi 3 Continuous, 1 category\n\tchartil.plot(heart_disease_df, ['chol', 'thalach', 'trestbps', 'target']) # Group Color highlighted 3D plot\n\n\n# Multi 3 category, 2 Continuous\n\tchartil.plot(heart_disease_df, ['sex','cp','target','thalach','trestbps']) # Paired scatter plot\n\n\n# Full working demo available on [kaggle here](https://www.kaggle.com/keshavshetty/chart-util-chartil)\n\nAuto Linear Regression (Click to expand)
\n\n# Auto Linear Regression\n\n#### We have seen Auto ML like H2O which is a blackbox approach to generate models. \nDuring our model building process, we try with brute force/TrialnError/several combinations to come up with best model. \nHowever trying these possibilities manually is a laborious process.\nIn order to overcome or atleast have a base model automatically I developed this auto linear regression using backward feature elimination technique.\n\nThe library/package can be found [here](https://pypi.org/project/kesh-utils/) and source code [here](https://github.com/KeshavShetty/ds/tree/master/KUtils/linear_regression)\n\n# How Auto LR works?\n\nWe throw the cleaned dataset to autolr.fit(<>)\nThe method will \n- Treat categorical variable if applicable(dummy creation/One hot encoding)\n- First model - Run the RFE on dataset\n- For remaining features elimination - it follows backward elimination - one feature at a time\n - combination of vif and p-values of coefficients (Eliminate with higher vif and p-value combination\n - vif only (or eliminate one with higher vif)\n - p-values only (or eliminate one with higher p-value)\n- Everytime when a feature is identified we build new model and repeat the process\n- on every iteration if adjusted R2 affected significantly, we re-add/retain it and select next possible feature to eliminate.\n- Repeat until program can't proceed further with above logic.\n\n# Auto Linear Regression Package/Function details\n\nThe method autolr.fit() has below parameters\n- df, (The full dataframe)\n- dependent_column, (Target column)\n- p_value_cutoff = 0.01, (Threashold p-values of features to use while filtering features during backward elimination step, Default 0.01)\n- vif_cutoff = 5, (Threashold co-relation of vif values of features to use while filtering features during backward elimination step, Default 5)\n- acceptable_r2_change = 0.02, (Restrict degradtion of model efficiency by controlling loss of change in R2, Default 0.02)\n- scale_numerical = False, (Flag to convert/scale numerical fetures using StandardScaler)\n- include_target_column_from_scaling = True, (Flag to indiacte weather to include target column from scaling)\n- dummies_creation_drop_column_preference='dropFirst', (Available options dropFirst, dropMax, dropMin - While creating dummies which clum drop to convert to one hot)\n- train_split_size = 0.7, (Train/Test split ration to be used)\n- max_features_to_select = 0, (Set the number of features to be qualified from RFE before entring auto backward elimination)\n- random_state_to_use=100, (Self explanatory)\n- include_data_in_return = False, (Include the data generated/used in Auto LR which might have gobne thru scaling, dummy creation etc.)\n- verbose=False (Enable to print detailed debug messgaes)\n\nAbove method returns 'model_info' dictionary which will have all the details used while performing auto fit. \n\n# Full working demo available on [kaggle here](https://www.kaggle.com/keshavshetty/auto-linear-regression)\nClustered Linear Regression (Click to expand)
\n\n# Clustered Linear Regression\n\nFor a linear regression approach we try to fit a best model on entire dataset. \nHowever often we have seen within dataset based on a particular feature the dataset behaves totally different and single model is not the best solutions, \ninstead have multiple model which applied on different subset or filtered data does better.\n\n\n# How to find the feature which splits the dataset into multiple sub dataset (and there after build and apply different models)\n\nThere is no easy solution, instead use trial and error or brute force to subset data on different feature and build multiple model. \nThis clustred or grouped Linear Regression does the same. \nYou send the entire dataset and specifiy list of columns to separate the dataset individually and return the kpi measures like rmse or r2 etc and then decide which way to go.\n\n\n# How \"Clustered Linear Regression\" works?\n\n- First it lists possible combinations \n- For each possible combinations split the data into subset\n- For each subset execute the Auto Linear Regression. Check previous kaggle post on this.\n- Return summary or consolidated kpi measures at group level.\n\n# The API clustlr.fit() has below parameters\n\n- data_df (Full dataset)\n- feature_group_list (List of column on which filter and group the data\n- dependent_column (The target column)\n- max_level = 2 (When it is 2 it uses two feature combination to filter)\n- min_leaf_in_filtered_dataset=1000 (Condition the minimum datapoints in subgroup without which autolr will not be executed)\n- no_of_bins_for_continuous_feature=10 (number of bins to be created when you use continuous varibale for grouping)\n- verbose (Use True if you want detailed debug/log message)\n\n# Full working demo available on [kaggle here](https://www.kaggle.com/keshavshetty/clustered-linear-regression)\n\nAuto Logistic Regression (Click to expand)
\n\n# Auto Logistic Regression\n\n