Harjot Singh

“If it’s true what is said, that only the wise discover the wise, then it must also be true that the lone wolf symbolizes either the biggest fool on the planet or the biggest Einstein on the planet.” 

― Criss Jami, Diotima, Battery, Electric Personality

What is Hypothesis?
It is a premise or claim that we want to test.

Null Hypothesis – Ho – Currently accepted value for a parameter.
Alternative Hypothesis (Research Hypothesis) – Ha – Claim to be tested.

Let’s understand it with an example.

Example – It is believed that a candy machine makes chocolate bars that are on average 5g. A worker claims that the machine after maintenance no longer makes 5g bars. What would be Ho and Ha here?

Ho: μ = 5g
Ha: μ ≠ 5g

• Ho and Ha are mathematical opposites.
• You assume null hypothesis to be true unless evidence points otherwise.

Possible Outcome of this test:
– Reject the null hypothesis.
– Fail to reject Null Hypothesis.

Next is, How do we do that?

Test Statistic – calculated from sample data, used to decide.
Let’s continue to understand it with our example above.

We sample 50 chocolate bars (not practical to take all the bars produced).
– Get Average Value for 50 bars.
– use this information to calculate test statistic.

What do we mean when we say ‘statistically significant’ ?
– It is where do we draw the line to make a decision.

Continuing…
Let’s say the average of the sample of 50 bars comes out to be 5.12 g (sampled on Monday), 5.72 g (sampled on wednesday), 7.23 g (sampled on Friday)
Avg: 5.12 g, Avg: 5.72 g, Avg: 7.23g

Now most of us will see these averages and form different opinions . Some people might say we should reject the null hypothesis based on third sample which averages 7.23. Some might say we can accept the null hypothesis based om first sample which averages 5.12 g.

But there is no concreteness here. We’re all talking here.

Statistics is not about how you think it should be. We have to have a concrete way, looking at the null hypothesis, collecting the data and having a concrete method to decide when we accept the null hypothesis and when we leave it there.

And that is what a hypothesis test does!

A hypothesis test collects a data, put it in a equation, get a number back, and that number is going to show you how you decide when that test statistic is too high or too low, and when you reject a null hypothesis and when you don’t ( based on concrete boundaries).

Level of confidence (LOC) : C – 95 %, 99%
It is how confident are we in our decision.

Don’t forget we are doing a hypothesis test. We are testing something and we are deciding to reject the null hypothesis, or to fail to reject a null hypothesis. The level of confidence is telling us how sure we are that we did the right thing (rejecting or failing to reject the null hypothesis).

Level of Significance :
denoted by ‘α’.
α = 1 – C
if LOC = 95 %, C = 0.95
the, α = 1 – 0.95 = 0.05

YOU DON’T HAVE TO PROVE THAT NULL HYPOTHESIS IS TRUE, HYPOTHESIS TESTING ALREADY ASSUMES NULL HYPOTHESIS TO BE TRUE. YOU EITHER REJECT A NULL HYPOTHESIS OR FAIL TO REJECT A NULL HYPOTHESIS.
This is what statistics do.

In statistics, kernel density estimation (KDE) is a non-parametric way to estimate the probability density function (PDF) of a random variable.
Kernel density estimation is a fundamental data smoothing problem where inferences about the population are made, based on a finite data sample.

Kernel density estimation is a really useful statistical tool with an intimidating name. Often shortened to KDE, it’s a technique that let’s you create a smooth curve given a set of data.

This can be useful if you want to visualize just the “shape” of some data, as a kind of continuous replacement for the discrete histogram.

How does it work?

The KDE algorithm takes a parameter, bandwidth, that affects how “smooth” the resulting curve is. 

Changing the bandwidth changes the shape of the kernel: a lower bandwidth means only points very close to the current position are given any weight, which leads to the estimate looking squiggly; a higher bandwidth means a shallow kernel where distant points can contribute.

What is a kernel, in non-parametric statistics?

In non-parametric statistics, a kernel is a weighting function used in non-parametric estimation techniques. Kernels are used in kernel density estimation to estimate random variables‘ density functions, or in kernel regression to estimate the conditional expectation of a random variable.

(https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1m)XWo6uco/wiki/Kernel_(statistics).html

How to plot KDE using Pandas Series?

Let’s see..

Series.plot.kde(bw_method=None, ind=None, **kwds)

Parameters:

• bw_method

The method used to calculate the estimator bandwidth. This can be ‘scott’, ‘silverman’, a scalar constant or a callable. If None (default), ‘scott’ is used.

• ind

Evaluation points for the estimated PDF :
If None (default), 1000 equally spaced points are used.
If ind is a NumPy array, the KDE is evaluated at the points passed. If ind is an integer, ind number of equally spaced points are used.

Returns :

axes : matplotlib.axes.Axes or numpy.ndarray of them

Note -  A scalar bandwidth can be specified. Using a small bandwidth value can lead to over-fitting, while using a large bandwidth value may result in under-fitting: 

Happy to help 🙂 You can reach out to me at harjotsaini69@gmail.com for any questions.

Happy Learning!!!