/Python/Using pandas stack and subset to return a dataframe object of highly correated pairs

Using pandas stack and subset to return a dataframe object of highly correated pairs

# save absolute value of correlation matrix as a data frame
# converts all values to absolute value
# stacks the row:column pairs into a multindex
# reset the index to set the multindex to seperate columns
# sort values. 0 is the column automatically generated by the stacking

df= old_df.corr().abs().stack().reset_index().sort_values(0, ascending=False)

# zip the variable name columns (Which were only named level_0 and level_1 by default) 
# in a new column named "pairs".
df['pairs'] = list(zip(df.level_0, df.level_1))

# set index to pairs
df.set_index(['pairs'], inplace = True)

#d rop level columns
df.drop(columns=['level_1', 'level_0'], inplace = True)

# rename correlation column as cc rather than 0
df.columns = ['cc']

# drop duplicates. This could be dangerous if you have variables perfectly correlated with variables other than themselves.
# for the sake of exercise, kept it in.
df.drop_duplicates(inplace=True)

# subset to include desired range
df[(df.cc>.75) & (df.cc <1)]

Similar codes

1 vote136 views1 answer
Number of rows or columns in numpy ndarray python
Python
0 vote145 views1 answer
Install pip3 for python 3.7
Python
1 vote195 views2 answers
Redirect to the same page django
Python Django
81 votes139 views6 answers
Python write to file
Python
17 votes138 views3 answers
Python all possible combinations of multiple lists
Python
0 vote143 views3 answers
In is_lodes_form( : Missing id-axis pairings.
Python
1 vote130 views1 answer
Pyhton find dates in weeks
Python
1 vote150 views1 answer
How to convert int in python
Python
68 votes143 views6 answers
Python if __main__
Python
6 votes110 views1 answer
Unicodeencodeerror: 'charmap' codec can't encode characters in position 0-3: character maps to <undefined>
Python