4.5 Patterns, trends and relationships

Key points

1. Correlation measures how two variables move together. Causation needs more evidence, such as a controlled experiment.

   What NVIDIA says (2)

   “pearson : Standard correlation coefficient spearman : Spearman rank correlation”

   — cuDF API: DataFrame.corr

   “However, it’s important to remember that correlation and causation are two different things.”

   — NVIDIA Glossary: Linear Regression and Logistic Regression

2. Monotonic means one variable tends to rise when the other rises, not always at the same rate. Spearman uses ranks, so it captures that without assuming a line.

   What NVIDIA says (1)

   “Method used to compute correlation: pearson : Standard correlation coefficient spearman : Spearman rank correlation”

   — cuDF API: DataFrame.corr

3. A trend is the long-run direction of a series. A rolling window averages nearby values to smooth out noise.

   What NVIDIA says (1)

   “Parameters : window int, offset or a BaseIndexer subclass Size of the window, i.e., the number of observations used to calculate the statistic.”

   — cuDF API: DataFrame.rolling

4. A linear model assumes a straight-line relationship. When the pattern curves, a nonlinear model can describe it better.

   What NVIDIA says (2)

   “While there is a strong relationship between population and time, the relationship is not linear because various factors influence changes from year to year.”

   — NVIDIA Glossary: Linear Regression and Logistic Regression

   “Nonlinear regression can estimate models with arbitrary relationships between independent and dependent variables.”

   — NVIDIA Glossary: Linear Regression and Logistic Regression

Key terms

• Correlation: A measure of how two variables move together; it does not prove cause.
• Rolling mean: The average over a sliding window of recent rows, used to smooth noise.

Sample question

Two columns have a strong Pearson correlation. What can you conclude?