Week 7 Wednesday

Announcements

• In discussion tomorrow, Jinghao will introduce some important new material: the MNIST dataset of handwritten digits.

• My mistake at the end of Monday’s class was hard to spot: I rounded the numbers too much before plugging in to the sigmoid function.

import pandas as pd
import numpy as np
import altair as alt
import seaborn as sns

Recap of Monday (and a correction)

On Monday we used Logistic Regression with the flipper length and bill length columns in the penguins dataset to predict if a penguin is in the Chinstrap species.

df = sns.load_dataset("penguins").dropna(axis=0)
df["isChinstrap"] = (df["species"] == "Chinstrap")

cols = ["flipper_length_mm", "bill_length_mm"]

alt.Chart(df).mark_circle().encode(
    x=alt.X(cols[0], scale=alt.Scale(zero=False)),
    y=alt.Y(cols[1], scale=alt.Scale(zero=False)),
    color="species"
)

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression()

clf.fit(df[cols], df["isChinstrap"])

LogisticRegression()

df["pred"] = clf.predict(df[cols])

The Chinstrap penguins all appear in the same part of the dataset, so it’s not surprising that none of these three consecutive rows is a Chinstrap penguin. The most important thing to notice is that the wrong prediction was made for the row with label 19.

df.loc[[18, 19, 20], cols + ["isChinstrap", "pred"]]

	flipper_length_mm	bill_length_mm	isChinstrap	pred
18	184.0	34.4	False	False
19	194.0	46.0	False	True
20	174.0	37.8	False	False

If you look one cell further down, you see that the classes are listed in the order False then True. In this NumPy array of predicted probabilities, the left column corresponds to False and the right column corresponds to True. Notice how much more confident the model is for row 18 than for row 19. That is good, because the model was correct for row 18 and incorrect for row 19.

# predicted probabilities
clf.predict_proba(df.loc[[18, 19, 20], cols])

array([[9.99771697e-01, 2.28303114e-04],
       [3.29559670e-01, 6.70440330e-01],
       [7.71818762e-01, 2.28181238e-01]])

The order of the classes listed in the classes_ attribute is the same as the order for the columns above.

clf.classes_

array([False,  True])

At the end of class Monday, I tried to recover the 0.67044 number for the predicted probability that the penguin in the row with label 19 is a Chinstrap penguin.

clf.coef_

array([[-0.34802208,  1.08405225]])

clf.intercept_

array([18.36005773])

df.loc[19, cols]

flipper_length_mm    194.0
bill_length_mm        46.0
Name: 19, dtype: object

Be sure you see where this formula comes from, in terms of the information above.

# Why not 0.67 or maybe 1-0.67???
1/(1+np.exp(-(18.36+-0.35*194+1.08*46)))

0.5349429451582182

It turned out that I wasn’t using enough decimal places when I plugged in the numbers. Here we see the 0.67 we expected. All I did was replace -0.35 with -0.348 and replace 1.08 with 1.084.

# One more digit of precision to my input coefficients
1/(1+np.exp(-(18.36+-0.348*194+1.084*46)))

0.670842935992734

Notice above that the coef_ attribute is two-dimensional. In this case, it’s not clear why that is, but below when we have three target classes (rather than two), it will be more clear why we have this extra dimension.

clf.coef_[0]

array([-0.34802208,  1.08405225])

Here we are using the exact coefficients. This result should be the exact predicted probability. (Up to possible numerical precision issues related to floating point numbers.)

# more precision
1/(1+np.exp(-(18.36+clf.coef_[0][0]*194+clf.coef_[0][1]*46)))

0.6704275750208124

Interpretation question:

• As flipper length increases, is the penguin more or less likely to be a Chinstrap penguin, according to our model. What about bill length?

If you look at the overall formula, this will get pretty confusing. Better is to notice that \(\sigma(x) = 1/(1+e^{-x})\) is an increasing function, and our formula is \(\sigma(L(x))\) for some linear term \(L(x)\). This interpretation question is much easier when we focus on the linear part: all we need to do is look at the signs of the coefficients.

Answer: as flipper length increases, the probability of being Chinstrap decreases. Why? -0.34 is negative. As bill length increases, probability increases (1.08 is positive).

Decision boundary

• Drop the “pred” column from df (we will make a new one below) using the drop method and a suitable axis keyword argument.

We are using axis=1 because it is the column labels that are changing (one of the column labels is being removed). I added errors="ignore" so that if we execute this cell twice, no error is raised.

df = df.drop("pred", axis=1, errors="ignore")

• Fit a new logistic regression classifier, using the same input features, but this time using the “species” column as our target. (This is our first time seeing logistic regression with more than two classes. When we perform classification with two classes, it is called “Binary Classification”, and is often easier to explain.)

Even though we have three output classes, the procedure is the same we have been using all along.

clf = LogisticRegression()

The classifier will automatically report outputs using the same names that are in the “species” column.

clf.fit(df[cols], df["species"])

LogisticRegression()

Remember how the order of False and True above was important. Here the order of the penguin species is also important. I think they will always be in alphabetical order (but I’m not certain of that).

clf.classes_

array(['Adelie', 'Chinstrap', 'Gentoo'], dtype=object)

• Check the coef_ attribute. How does it relate to the coef_ attribute we found above, where we were only considering Chinstrap penguins?

clf.coef_

array([[-0.10544131, -0.59253116],
       [-0.2674632 ,  0.71305121],
       [ 0.37290451, -0.12052006]])

Here are the predicted probabilities for that “incorrect” row 19 from above. The model is actually a little more confident in this case (and is still wrong), it thinks there is a 91.5% chance the penguin is a Chinstrap penguin. (We know the middle number corresponds to Chinstrap, by looking at the order of the values in clf.classes_.)

clf.predict_proba(df.loc[[19], cols])

array([[0.08336009, 0.91516209, 0.00147781]])

Here is what it looks like if we evaluate the predict_proba method on a 4-row sub-DataFrame. Notice how the output has four rows also and three columns (one column for each target class).

clf.predict_proba(df.loc[[18, 19, 20, 21], cols])

array([[9.99984693e-01, 1.46859326e-05, 6.21326672e-07],
       [8.33600949e-02, 9.15162092e-01, 1.47781293e-03],
       [9.93753092e-01, 6.24688236e-03, 2.57127059e-08],
       [9.97917055e-01, 2.08251094e-03, 4.34432466e-07]])

• Add a column “pred” to df containing the predicted values.

This is no different from above.

df["pred"] = clf.predict(df[cols])

Notice how the “pred” column at the far right contains penguin species strings. So the predict method can output strings, not just numbers or Boolean values.

df

	species	island	bill_length_mm	bill_depth_mm	flipper_length_mm	body_mass_g	sex	isChinstrap	pred
0	Adelie	Torgersen	39.1	18.7	181.0	3750.0	Male	False	Adelie
1	Adelie	Torgersen	39.5	17.4	186.0	3800.0	Female	False	Adelie
2	Adelie	Torgersen	40.3	18.0	195.0	3250.0	Female	False	Adelie
4	Adelie	Torgersen	36.7	19.3	193.0	3450.0	Female	False	Adelie
5	Adelie	Torgersen	39.3	20.6	190.0	3650.0	Male	False	Adelie
...	...	...	...	...	...	...	...	...	...
338	Gentoo	Biscoe	47.2	13.7	214.0	4925.0	Female	False	Gentoo
340	Gentoo	Biscoe	46.8	14.3	215.0	4850.0	Female	False	Gentoo
341	Gentoo	Biscoe	50.4	15.7	222.0	5750.0	Male	False	Gentoo
342	Gentoo	Biscoe	45.2	14.8	212.0	5200.0	Female	False	Gentoo
343	Gentoo	Biscoe	49.9	16.1	213.0	5400.0	Male	False	Gentoo

333 rows × 9 columns

• Make an Altair scatter plot showing the predicted values.

The portion where the model switches from one prediction to another prediction is called the “decision boundary”. It’s hard to recognize the decision boundary in this picture, because there is so much empty space.

You should compare this to the true species values which were plotted at the very top of this notebook. The predicted species values are very close, but a little more regular, than the actual values.

alt.Chart(df).mark_circle().encode(
    x=alt.X(cols[0], scale=alt.Scale(zero=False)),
    y=alt.Y(cols[1], scale=alt.Scale(zero=False)),
    color="pred"
)

It is a little hard to see from the above picture how the predictions are made. It turns out there are a few straight line segments, and on one side of each line segment, one prediction is made, and on the other side, another prediction is made. We will make a fake dataset from which these “decision boundaries” are more clear.

• Using np.linspace, make a NumPy array of 70 equally spaced x-coordinates and 70 equally spaced y-coordinates. Name these NumPy arrays x and y.

Notice how the ranges chosen here are chosen so that it matches the approximate ranges of the flipper length and the bill length.

x = np.linspace(170, 235, 70)
y = np.linspace(30, 60, 70)

• Make a DataFrame df_art (for “artificial”) containing all the possible pairs of coordinates from x and y. (We chose 70 above so df_art will have 4900 rows, which is a good length for Altair.)

To get these pairs, we can for example use the NumPy function meshgrid or we can use itertools.product. (In Worksheet 14, things are a little easier, we just choose random values, and don’t worry about making them evenly spaced.)

• Add a corresponding “pred” column to df_art.

We start out here using the product function from the itertools library.

from itertools import product

Think of product as formuing something like a Cartesian product. That’s not clear at all from looking at it.

product(x,y)

<itertools.product at 0x7f6843e4df00>

If we convert it to a list, it’s more clear.

list(product(x,y))

[(170.0, 30.0),
 (170.0, 30.434782608695652),
 (170.0, 30.869565217391305),
 (170.0, 31.304347826086957),
 (170.0, 31.73913043478261),
 (170.0, 32.17391304347826),
 (170.0, 32.608695652173914),
 (170.0, 33.04347826086956),
 (170.0, 33.47826086956522),
 (170.0, 33.91304347826087),
 (170.0, 34.34782608695652),
 (170.0, 34.78260869565217),
 (170.0, 35.21739130434783),
 (170.0, 35.65217391304348),
 (170.0, 36.08695652173913),
 (170.0, 36.52173913043478),
 (170.0, 36.95652173913044),
 (170.0, 37.391304347826086),
 (170.0, 37.82608695652174),
 (170.0, 38.26086956521739),
 (170.0, 38.69565217391305),
 (170.0, 39.130434782608695),
 (170.0, 39.565217391304344),
 (170.0, 40.0),
 (170.0, 40.434782608695656),
 (170.0, 40.869565217391305),
 (170.0, 41.30434782608695),
 (170.0, 41.73913043478261),
 (170.0, 42.17391304347826),
 (170.0, 42.608695652173914),
 (170.0, 43.04347826086956),
 (170.0, 43.47826086956522),
 (170.0, 43.91304347826087),
 (170.0, 44.34782608695652),
 (170.0, 44.78260869565217),
 (170.0, 45.21739130434783),
 (170.0, 45.65217391304348),
 (170.0, 46.086956521739125),
 (170.0, 46.52173913043478),
 (170.0, 46.95652173913044),
 (170.0, 47.391304347826086),
 (170.0, 47.826086956521735),
 (170.0, 48.26086956521739),
 (170.0, 48.69565217391305),
 (170.0, 49.130434782608695),
 (170.0, 49.565217391304344),
 (170.0, 50.0),
 (170.0, 50.434782608695656),
 (170.0, 50.869565217391305),
 (170.0, 51.30434782608695),
 (170.0, 51.73913043478261),
 (170.0, 52.173913043478265),
 (170.0, 52.608695652173914),
 (170.0, 53.04347826086956),
 (170.0, 53.47826086956522),
 (170.0, 53.91304347826087),
 (170.0, 54.347826086956516),
 (170.0, 54.78260869565217),
 (170.0, 55.21739130434783),
 (170.0, 55.65217391304348),
 (170.0, 56.086956521739125),
 (170.0, 56.52173913043478),
 (170.0, 56.95652173913044),
 (170.0, 57.391304347826086),
 (170.0, 57.826086956521735),
 (170.0, 58.26086956521739),
 (170.0, 58.69565217391305),
 (170.0, 59.130434782608695),
 (170.0, 59.565217391304344),
 (170.0, 60.0),
 (170.94202898550725, 30.0),
 (170.94202898550725, 30.434782608695652),
 (170.94202898550725, 30.869565217391305),
 (170.94202898550725, 31.304347826086957),
 (170.94202898550725, 31.73913043478261),
 (170.94202898550725, 32.17391304347826),
 (170.94202898550725, 32.608695652173914),
 (170.94202898550725, 33.04347826086956),
 (170.94202898550725, 33.47826086956522),
 (170.94202898550725, 33.91304347826087),
 (170.94202898550725, 34.34782608695652),
 (170.94202898550725, 34.78260869565217),
 (170.94202898550725, 35.21739130434783),
 (170.94202898550725, 35.65217391304348),
 (170.94202898550725, 36.08695652173913),
 (170.94202898550725, 36.52173913043478),
 (170.94202898550725, 36.95652173913044),
 (170.94202898550725, 37.391304347826086),
 (170.94202898550725, 37.82608695652174),
 (170.94202898550725, 38.26086956521739),
 (170.94202898550725, 38.69565217391305),
 (170.94202898550725, 39.130434782608695),
 (170.94202898550725, 39.565217391304344),
 (170.94202898550725, 40.0),
 (170.94202898550725, 40.434782608695656),
 (170.94202898550725, 40.869565217391305),
 (170.94202898550725, 41.30434782608695),
 (170.94202898550725, 41.73913043478261),
 (170.94202898550725, 42.17391304347826),
 (170.94202898550725, 42.608695652173914),
 (170.94202898550725, 43.04347826086956),
 (170.94202898550725, 43.47826086956522),
 (170.94202898550725, 43.91304347826087),
 (170.94202898550725, 44.34782608695652),
 (170.94202898550725, 44.78260869565217),
 (170.94202898550725, 45.21739130434783),
 (170.94202898550725, 45.65217391304348),
 (170.94202898550725, 46.086956521739125),
 (170.94202898550725, 46.52173913043478),
 (170.94202898550725, 46.95652173913044),
 (170.94202898550725, 47.391304347826086),
 (170.94202898550725, 47.826086956521735),
 (170.94202898550725, 48.26086956521739),
 (170.94202898550725, 48.69565217391305),
 (170.94202898550725, 49.130434782608695),
 (170.94202898550725, 49.565217391304344),
 (170.94202898550725, 50.0),
 (170.94202898550725, 50.434782608695656),
 (170.94202898550725, 50.869565217391305),
 (170.94202898550725, 51.30434782608695),
 (170.94202898550725, 51.73913043478261),
 (170.94202898550725, 52.173913043478265),
 (170.94202898550725, 52.608695652173914),
 (170.94202898550725, 53.04347826086956),
 (170.94202898550725, 53.47826086956522),
 (170.94202898550725, 53.91304347826087),
 (170.94202898550725, 54.347826086956516),
 (170.94202898550725, 54.78260869565217),
 (170.94202898550725, 55.21739130434783),
 (170.94202898550725, 55.65217391304348),
 (170.94202898550725, 56.086956521739125),
 (170.94202898550725, 56.52173913043478),
 (170.94202898550725, 56.95652173913044),
 (170.94202898550725, 57.391304347826086),
 (170.94202898550725, 57.826086956521735),
 (170.94202898550725, 58.26086956521739),
 (170.94202898550725, 58.69565217391305),
 (170.94202898550725, 59.130434782608695),
 (170.94202898550725, 59.565217391304344),
 (170.94202898550725, 60.0),
 (171.8840579710145, 30.0),
 (171.8840579710145, 30.434782608695652),
 (171.8840579710145, 30.869565217391305),
 (171.8840579710145, 31.304347826086957),
 (171.8840579710145, 31.73913043478261),
 (171.8840579710145, 32.17391304347826),
 (171.8840579710145, 32.608695652173914),
 (171.8840579710145, 33.04347826086956),
 (171.8840579710145, 33.47826086956522),
 (171.8840579710145, 33.91304347826087),
 (171.8840579710145, 34.34782608695652),
 (171.8840579710145, 34.78260869565217),
 (171.8840579710145, 35.21739130434783),
 (171.8840579710145, 35.65217391304348),
 (171.8840579710145, 36.08695652173913),
 (171.8840579710145, 36.52173913043478),
 (171.8840579710145, 36.95652173913044),
 (171.8840579710145, 37.391304347826086),
 (171.8840579710145, 37.82608695652174),
 (171.8840579710145, 38.26086956521739),
 (171.8840579710145, 38.69565217391305),
 (171.8840579710145, 39.130434782608695),
 (171.8840579710145, 39.565217391304344),
 (171.8840579710145, 40.0),
 (171.8840579710145, 40.434782608695656),
 (171.8840579710145, 40.869565217391305),
 (171.8840579710145, 41.30434782608695),
 (171.8840579710145, 41.73913043478261),
 (171.8840579710145, 42.17391304347826),
 (171.8840579710145, 42.608695652173914),
 (171.8840579710145, 43.04347826086956),
 (171.8840579710145, 43.47826086956522),
 (171.8840579710145, 43.91304347826087),
 (171.8840579710145, 44.34782608695652),
 (171.8840579710145, 44.78260869565217),
 (171.8840579710145, 45.21739130434783),
 (171.8840579710145, 45.65217391304348),
 (171.8840579710145, 46.086956521739125),
 (171.8840579710145, 46.52173913043478),
 (171.8840579710145, 46.95652173913044),
 (171.8840579710145, 47.391304347826086),
 (171.8840579710145, 47.826086956521735),
 (171.8840579710145, 48.26086956521739),
 (171.8840579710145, 48.69565217391305),
 (171.8840579710145, 49.130434782608695),
 (171.8840579710145, 49.565217391304344),
 (171.8840579710145, 50.0),
 (171.8840579710145, 50.434782608695656),
 (171.8840579710145, 50.869565217391305),
 (171.8840579710145, 51.30434782608695),
 (171.8840579710145, 51.73913043478261),
 (171.8840579710145, 52.173913043478265),
 (171.8840579710145, 52.608695652173914),
 (171.8840579710145, 53.04347826086956),
 (171.8840579710145, 53.47826086956522),
 (171.8840579710145, 53.91304347826087),
 (171.8840579710145, 54.347826086956516),
 (171.8840579710145, 54.78260869565217),
 (171.8840579710145, 55.21739130434783),
 (171.8840579710145, 55.65217391304348),
 (171.8840579710145, 56.086956521739125),
 (171.8840579710145, 56.52173913043478),
 (171.8840579710145, 56.95652173913044),
 (171.8840579710145, 57.391304347826086),
 (171.8840579710145, 57.826086956521735),
 (171.8840579710145, 58.26086956521739),
 (171.8840579710145, 58.69565217391305),
 (171.8840579710145, 59.130434782608695),
 (171.8840579710145, 59.565217391304344),
 (171.8840579710145, 60.0),
 (172.82608695652175, 30.0),
 (172.82608695652175, 30.434782608695652),
 (172.82608695652175, 30.869565217391305),
 (172.82608695652175, 31.304347826086957),
 (172.82608695652175, 31.73913043478261),
 (172.82608695652175, 32.17391304347826),
 (172.82608695652175, 32.608695652173914),
 (172.82608695652175, 33.04347826086956),
 (172.82608695652175, 33.47826086956522),
 (172.82608695652175, 33.91304347826087),
 (172.82608695652175, 34.34782608695652),
 (172.82608695652175, 34.78260869565217),
 (172.82608695652175, 35.21739130434783),
 (172.82608695652175, 35.65217391304348),
 (172.82608695652175, 36.08695652173913),
 (172.82608695652175, 36.52173913043478),
 (172.82608695652175, 36.95652173913044),
 (172.82608695652175, 37.391304347826086),
 (172.82608695652175, 37.82608695652174),
 (172.82608695652175, 38.26086956521739),
 (172.82608695652175, 38.69565217391305),
 (172.82608695652175, 39.130434782608695),
 (172.82608695652175, 39.565217391304344),
 (172.82608695652175, 40.0),
 (172.82608695652175, 40.434782608695656),
 (172.82608695652175, 40.869565217391305),
 (172.82608695652175, 41.30434782608695),
 (172.82608695652175, 41.73913043478261),
 (172.82608695652175, 42.17391304347826),
 (172.82608695652175, 42.608695652173914),
 (172.82608695652175, 43.04347826086956),
 (172.82608695652175, 43.47826086956522),
 (172.82608695652175, 43.91304347826087),
 (172.82608695652175, 44.34782608695652),
 (172.82608695652175, 44.78260869565217),
 (172.82608695652175, 45.21739130434783),
 (172.82608695652175, 45.65217391304348),
 (172.82608695652175, 46.086956521739125),
 (172.82608695652175, 46.52173913043478),
 (172.82608695652175, 46.95652173913044),
 (172.82608695652175, 47.391304347826086),
 (172.82608695652175, 47.826086956521735),
 (172.82608695652175, 48.26086956521739),
 (172.82608695652175, 48.69565217391305),
 (172.82608695652175, 49.130434782608695),
 (172.82608695652175, 49.565217391304344),
 (172.82608695652175, 50.0),
 (172.82608695652175, 50.434782608695656),
 (172.82608695652175, 50.869565217391305),
 (172.82608695652175, 51.30434782608695),
 (172.82608695652175, 51.73913043478261),
 (172.82608695652175, 52.173913043478265),
 (172.82608695652175, 52.608695652173914),
 (172.82608695652175, 53.04347826086956),
 (172.82608695652175, 53.47826086956522),
 (172.82608695652175, 53.91304347826087),
 (172.82608695652175, 54.347826086956516),
 (172.82608695652175, 54.78260869565217),
 (172.82608695652175, 55.21739130434783),
 (172.82608695652175, 55.65217391304348),
 (172.82608695652175, 56.086956521739125),
 (172.82608695652175, 56.52173913043478),
 (172.82608695652175, 56.95652173913044),
 (172.82608695652175, 57.391304347826086),
 (172.82608695652175, 57.826086956521735),
 (172.82608695652175, 58.26086956521739),
 (172.82608695652175, 58.69565217391305),
 (172.82608695652175, 59.130434782608695),
 (172.82608695652175, 59.565217391304344),
 (172.82608695652175, 60.0),
 (173.768115942029, 30.0),
 (173.768115942029, 30.434782608695652),
 (173.768115942029, 30.869565217391305),
 (173.768115942029, 31.304347826086957),
 (173.768115942029, 31.73913043478261),
 (173.768115942029, 32.17391304347826),
 (173.768115942029, 32.608695652173914),
 (173.768115942029, 33.04347826086956),
 (173.768115942029, 33.47826086956522),
 (173.768115942029, 33.91304347826087),
 (173.768115942029, 34.34782608695652),
 (173.768115942029, 34.78260869565217),
 (173.768115942029, 35.21739130434783),
 (173.768115942029, 35.65217391304348),
 (173.768115942029, 36.08695652173913),
 (173.768115942029, 36.52173913043478),
 (173.768115942029, 36.95652173913044),
 (173.768115942029, 37.391304347826086),
 (173.768115942029, 37.82608695652174),
 (173.768115942029, 38.26086956521739),
 (173.768115942029, 38.69565217391305),
 (173.768115942029, 39.130434782608695),
 (173.768115942029, 39.565217391304344),
 (173.768115942029, 40.0),
 (173.768115942029, 40.434782608695656),
 (173.768115942029, 40.869565217391305),
 (173.768115942029, 41.30434782608695),
 (173.768115942029, 41.73913043478261),
 (173.768115942029, 42.17391304347826),
 (173.768115942029, 42.608695652173914),
 (173.768115942029, 43.04347826086956),
 (173.768115942029, 43.47826086956522),
 (173.768115942029, 43.91304347826087),
 (173.768115942029, 44.34782608695652),
 (173.768115942029, 44.78260869565217),
 (173.768115942029, 45.21739130434783),
 (173.768115942029, 45.65217391304348),
 (173.768115942029, 46.086956521739125),
 (173.768115942029, 46.52173913043478),
 (173.768115942029, 46.95652173913044),
 (173.768115942029, 47.391304347826086),
 (173.768115942029, 47.826086956521735),
 (173.768115942029, 48.26086956521739),
 (173.768115942029, 48.69565217391305),
 (173.768115942029, 49.130434782608695),
 (173.768115942029, 49.565217391304344),
 (173.768115942029, 50.0),
 (173.768115942029, 50.434782608695656),
 (173.768115942029, 50.869565217391305),
 (173.768115942029, 51.30434782608695),
 (173.768115942029, 51.73913043478261),
 (173.768115942029, 52.173913043478265),
 (173.768115942029, 52.608695652173914),
 (173.768115942029, 53.04347826086956),
 (173.768115942029, 53.47826086956522),
 (173.768115942029, 53.91304347826087),
 (173.768115942029, 54.347826086956516),
 (173.768115942029, 54.78260869565217),
 (173.768115942029, 55.21739130434783),
 (173.768115942029, 55.65217391304348),
 (173.768115942029, 56.086956521739125),
 (173.768115942029, 56.52173913043478),
 (173.768115942029, 56.95652173913044),
 (173.768115942029, 57.391304347826086),
 (173.768115942029, 57.826086956521735),
 (173.768115942029, 58.26086956521739),
 (173.768115942029, 58.69565217391305),
 (173.768115942029, 59.130434782608695),
 (173.768115942029, 59.565217391304344),
 (173.768115942029, 60.0),
 (174.71014492753622, 30.0),
 (174.71014492753622, 30.434782608695652),
 (174.71014492753622, 30.869565217391305),
 (174.71014492753622, 31.304347826086957),
 (174.71014492753622, 31.73913043478261),
 (174.71014492753622, 32.17391304347826),
 (174.71014492753622, 32.608695652173914),
 (174.71014492753622, 33.04347826086956),
 (174.71014492753622, 33.47826086956522),
 (174.71014492753622, 33.91304347826087),
 (174.71014492753622, 34.34782608695652),
 (174.71014492753622, 34.78260869565217),
 (174.71014492753622, 35.21739130434783),
 (174.71014492753622, 35.65217391304348),
 (174.71014492753622, 36.08695652173913),
 (174.71014492753622, 36.52173913043478),
 (174.71014492753622, 36.95652173913044),
 (174.71014492753622, 37.391304347826086),
 (174.71014492753622, 37.82608695652174),
 (174.71014492753622, 38.26086956521739),
 (174.71014492753622, 38.69565217391305),
 (174.71014492753622, 39.130434782608695),
 (174.71014492753622, 39.565217391304344),
 (174.71014492753622, 40.0),
 (174.71014492753622, 40.434782608695656),
 (174.71014492753622, 40.869565217391305),
 (174.71014492753622, 41.30434782608695),
 (174.71014492753622, 41.73913043478261),
 (174.71014492753622, 42.17391304347826),
 (174.71014492753622, 42.608695652173914),
 (174.71014492753622, 43.04347826086956),
 (174.71014492753622, 43.47826086956522),
 (174.71014492753622, 43.91304347826087),
 (174.71014492753622, 44.34782608695652),
 (174.71014492753622, 44.78260869565217),
 (174.71014492753622, 45.21739130434783),
 (174.71014492753622, 45.65217391304348),
 (174.71014492753622, 46.086956521739125),
 (174.71014492753622, 46.52173913043478),
 (174.71014492753622, 46.95652173913044),
 (174.71014492753622, 47.391304347826086),
 (174.71014492753622, 47.826086956521735),
 (174.71014492753622, 48.26086956521739),
 (174.71014492753622, 48.69565217391305),
 (174.71014492753622, 49.130434782608695),
 (174.71014492753622, 49.565217391304344),
 (174.71014492753622, 50.0),
 (174.71014492753622, 50.434782608695656),
 (174.71014492753622, 50.869565217391305),
 (174.71014492753622, 51.30434782608695),
 (174.71014492753622, 51.73913043478261),
 (174.71014492753622, 52.173913043478265),
 (174.71014492753622, 52.608695652173914),
 (174.71014492753622, 53.04347826086956),
 (174.71014492753622, 53.47826086956522),
 (174.71014492753622, 53.91304347826087),
 (174.71014492753622, 54.347826086956516),
 (174.71014492753622, 54.78260869565217),
 (174.71014492753622, 55.21739130434783),
 (174.71014492753622, 55.65217391304348),
 (174.71014492753622, 56.086956521739125),
 (174.71014492753622, 56.52173913043478),
 (174.71014492753622, 56.95652173913044),
 (174.71014492753622, 57.391304347826086),
 (174.71014492753622, 57.826086956521735),
 (174.71014492753622, 58.26086956521739),
 (174.71014492753622, 58.69565217391305),
 (174.71014492753622, 59.130434782608695),
 (174.71014492753622, 59.565217391304344),
 (174.71014492753622, 60.0),
 (175.65217391304347, 30.0),
 (175.65217391304347, 30.434782608695652),
 (175.65217391304347, 30.869565217391305),
 (175.65217391304347, 31.304347826086957),
 (175.65217391304347, 31.73913043478261),
 (175.65217391304347, 32.17391304347826),
 (175.65217391304347, 32.608695652173914),
 (175.65217391304347, 33.04347826086956),
 (175.65217391304347, 33.47826086956522),
 (175.65217391304347, 33.91304347826087),
 (175.65217391304347, 34.34782608695652),
 (175.65217391304347, 34.78260869565217),
 (175.65217391304347, 35.21739130434783),
 (175.65217391304347, 35.65217391304348),
 (175.65217391304347, 36.08695652173913),
 (175.65217391304347, 36.52173913043478),
 (175.65217391304347, 36.95652173913044),
 (175.65217391304347, 37.391304347826086),
 (175.65217391304347, 37.82608695652174),
 (175.65217391304347, 38.26086956521739),
 (175.65217391304347, 38.69565217391305),
 (175.65217391304347, 39.130434782608695),
 (175.65217391304347, 39.565217391304344),
 (175.65217391304347, 40.0),
 (175.65217391304347, 40.434782608695656),
 (175.65217391304347, 40.869565217391305),
 (175.65217391304347, 41.30434782608695),
 (175.65217391304347, 41.73913043478261),
 (175.65217391304347, 42.17391304347826),
 (175.65217391304347, 42.608695652173914),
 (175.65217391304347, 43.04347826086956),
 (175.65217391304347, 43.47826086956522),
 (175.65217391304347, 43.91304347826087),
 (175.65217391304347, 44.34782608695652),
 (175.65217391304347, 44.78260869565217),
 (175.65217391304347, 45.21739130434783),
 (175.65217391304347, 45.65217391304348),
 (175.65217391304347, 46.086956521739125),
 (175.65217391304347, 46.52173913043478),
 (175.65217391304347, 46.95652173913044),
 (175.65217391304347, 47.391304347826086),
 (175.65217391304347, 47.826086956521735),
 (175.65217391304347, 48.26086956521739),
 (175.65217391304347, 48.69565217391305),
 (175.65217391304347, 49.130434782608695),
 (175.65217391304347, 49.565217391304344),
 (175.65217391304347, 50.0),
 (175.65217391304347, 50.434782608695656),
 (175.65217391304347, 50.869565217391305),
 (175.65217391304347, 51.30434782608695),
 (175.65217391304347, 51.73913043478261),
 (175.65217391304347, 52.173913043478265),
 (175.65217391304347, 52.608695652173914),
 (175.65217391304347, 53.04347826086956),
 (175.65217391304347, 53.47826086956522),
 (175.65217391304347, 53.91304347826087),
 (175.65217391304347, 54.347826086956516),
 (175.65217391304347, 54.78260869565217),
 (175.65217391304347, 55.21739130434783),
 (175.65217391304347, 55.65217391304348),
 (175.65217391304347, 56.086956521739125),
 (175.65217391304347, 56.52173913043478),
 (175.65217391304347, 56.95652173913044),
 (175.65217391304347, 57.391304347826086),
 (175.65217391304347, 57.826086956521735),
 (175.65217391304347, 58.26086956521739),
 (175.65217391304347, 58.69565217391305),
 (175.65217391304347, 59.130434782608695),
 (175.65217391304347, 59.565217391304344),
 (175.65217391304347, 60.0),
 (176.59420289855072, 30.0),
 (176.59420289855072, 30.434782608695652),
 (176.59420289855072, 30.869565217391305),
 (176.59420289855072, 31.304347826086957),
 (176.59420289855072, 31.73913043478261),
 (176.59420289855072, 32.17391304347826),
 (176.59420289855072, 32.608695652173914),
 (176.59420289855072, 33.04347826086956),
 (176.59420289855072, 33.47826086956522),
 (176.59420289855072, 33.91304347826087),
 (176.59420289855072, 34.34782608695652),
 (176.59420289855072, 34.78260869565217),
 (176.59420289855072, 35.21739130434783),
 (176.59420289855072, 35.65217391304348),
 (176.59420289855072, 36.08695652173913),
 (176.59420289855072, 36.52173913043478),
 (176.59420289855072, 36.95652173913044),
 (176.59420289855072, 37.391304347826086),
 (176.59420289855072, 37.82608695652174),
 (176.59420289855072, 38.26086956521739),
 (176.59420289855072, 38.69565217391305),
 (176.59420289855072, 39.130434782608695),
 (176.59420289855072, 39.565217391304344),
 (176.59420289855072, 40.0),
 (176.59420289855072, 40.434782608695656),
 (176.59420289855072, 40.869565217391305),
 (176.59420289855072, 41.30434782608695),
 (176.59420289855072, 41.73913043478261),
 (176.59420289855072, 42.17391304347826),
 (176.59420289855072, 42.608695652173914),
 (176.59420289855072, 43.04347826086956),
 (176.59420289855072, 43.47826086956522),
 (176.59420289855072, 43.91304347826087),
 (176.59420289855072, 44.34782608695652),
 (176.59420289855072, 44.78260869565217),
 (176.59420289855072, 45.21739130434783),
 (176.59420289855072, 45.65217391304348),
 (176.59420289855072, 46.086956521739125),
 (176.59420289855072, 46.52173913043478),
 (176.59420289855072, 46.95652173913044),
 (176.59420289855072, 47.391304347826086),
 (176.59420289855072, 47.826086956521735),
 (176.59420289855072, 48.26086956521739),
 (176.59420289855072, 48.69565217391305),
 (176.59420289855072, 49.130434782608695),
 (176.59420289855072, 49.565217391304344),
 (176.59420289855072, 50.0),
 (176.59420289855072, 50.434782608695656),
 (176.59420289855072, 50.869565217391305),
 (176.59420289855072, 51.30434782608695),
 (176.59420289855072, 51.73913043478261),
 (176.59420289855072, 52.173913043478265),
 (176.59420289855072, 52.608695652173914),
 (176.59420289855072, 53.04347826086956),
 (176.59420289855072, 53.47826086956522),
 (176.59420289855072, 53.91304347826087),
 (176.59420289855072, 54.347826086956516),
 (176.59420289855072, 54.78260869565217),
 (176.59420289855072, 55.21739130434783),
 (176.59420289855072, 55.65217391304348),
 (176.59420289855072, 56.086956521739125),
 (176.59420289855072, 56.52173913043478),
 (176.59420289855072, 56.95652173913044),
 (176.59420289855072, 57.391304347826086),
 (176.59420289855072, 57.826086956521735),
 (176.59420289855072, 58.26086956521739),
 (176.59420289855072, 58.69565217391305),
 (176.59420289855072, 59.130434782608695),
 (176.59420289855072, 59.565217391304344),
 (176.59420289855072, 60.0),
 (177.53623188405797, 30.0),
 (177.53623188405797, 30.434782608695652),
 (177.53623188405797, 30.869565217391305),
 (177.53623188405797, 31.304347826086957),
 (177.53623188405797, 31.73913043478261),
 (177.53623188405797, 32.17391304347826),
 (177.53623188405797, 32.608695652173914),
 (177.53623188405797, 33.04347826086956),
 (177.53623188405797, 33.47826086956522),
 (177.53623188405797, 33.91304347826087),
 (177.53623188405797, 34.34782608695652),
 (177.53623188405797, 34.78260869565217),
 (177.53623188405797, 35.21739130434783),
 (177.53623188405797, 35.65217391304348),
 (177.53623188405797, 36.08695652173913),
 (177.53623188405797, 36.52173913043478),
 (177.53623188405797, 36.95652173913044),
 (177.53623188405797, 37.391304347826086),
 (177.53623188405797, 37.82608695652174),
 (177.53623188405797, 38.26086956521739),
 (177.53623188405797, 38.69565217391305),
 (177.53623188405797, 39.130434782608695),
 (177.53623188405797, 39.565217391304344),
 (177.53623188405797, 40.0),
 (177.53623188405797, 40.434782608695656),
 (177.53623188405797, 40.869565217391305),
 (177.53623188405797, 41.30434782608695),
 (177.53623188405797, 41.73913043478261),
 (177.53623188405797, 42.17391304347826),
 (177.53623188405797, 42.608695652173914),
 (177.53623188405797, 43.04347826086956),
 (177.53623188405797, 43.47826086956522),
 (177.53623188405797, 43.91304347826087),
 (177.53623188405797, 44.34782608695652),
 (177.53623188405797, 44.78260869565217),
 (177.53623188405797, 45.21739130434783),
 (177.53623188405797, 45.65217391304348),
 (177.53623188405797, 46.086956521739125),
 (177.53623188405797, 46.52173913043478),
 (177.53623188405797, 46.95652173913044),
 (177.53623188405797, 47.391304347826086),
 (177.53623188405797, 47.826086956521735),
 (177.53623188405797, 48.26086956521739),
 (177.53623188405797, 48.69565217391305),
 (177.53623188405797, 49.130434782608695),
 (177.53623188405797, 49.565217391304344),
 (177.53623188405797, 50.0),
 (177.53623188405797, 50.434782608695656),
 (177.53623188405797, 50.869565217391305),
 (177.53623188405797, 51.30434782608695),
 (177.53623188405797, 51.73913043478261),
 (177.53623188405797, 52.173913043478265),
 (177.53623188405797, 52.608695652173914),
 (177.53623188405797, 53.04347826086956),
 (177.53623188405797, 53.47826086956522),
 (177.53623188405797, 53.91304347826087),
 (177.53623188405797, 54.347826086956516),
 (177.53623188405797, 54.78260869565217),
 (177.53623188405797, 55.21739130434783),
 (177.53623188405797, 55.65217391304348),
 (177.53623188405797, 56.086956521739125),
 (177.53623188405797, 56.52173913043478),
 (177.53623188405797, 56.95652173913044),
 (177.53623188405797, 57.391304347826086),
 (177.53623188405797, 57.826086956521735),
 (177.53623188405797, 58.26086956521739),
 (177.53623188405797, 58.69565217391305),
 (177.53623188405797, 59.130434782608695),
 (177.53623188405797, 59.565217391304344),
 (177.53623188405797, 60.0),
 (178.47826086956522, 30.0),
 (178.47826086956522, 30.434782608695652),
 (178.47826086956522, 30.869565217391305),
 (178.47826086956522, 31.304347826086957),
 (178.47826086956522, 31.73913043478261),
 (178.47826086956522, 32.17391304347826),
 (178.47826086956522, 32.608695652173914),
 (178.47826086956522, 33.04347826086956),
 (178.47826086956522, 33.47826086956522),
 (178.47826086956522, 33.91304347826087),
 (178.47826086956522, 34.34782608695652),
 (178.47826086956522, 34.78260869565217),
 (178.47826086956522, 35.21739130434783),
 (178.47826086956522, 35.65217391304348),
 (178.47826086956522, 36.08695652173913),
 (178.47826086956522, 36.52173913043478),
 (178.47826086956522, 36.95652173913044),
 (178.47826086956522, 37.391304347826086),
 (178.47826086956522, 37.82608695652174),
 (178.47826086956522, 38.26086956521739),
 (178.47826086956522, 38.69565217391305),
 (178.47826086956522, 39.130434782608695),
 (178.47826086956522, 39.565217391304344),
 (178.47826086956522, 40.0),
 (178.47826086956522, 40.434782608695656),
 (178.47826086956522, 40.869565217391305),
 (178.47826086956522, 41.30434782608695),
 (178.47826086956522, 41.73913043478261),
 (178.47826086956522, 42.17391304347826),
 (178.47826086956522, 42.608695652173914),
 (178.47826086956522, 43.04347826086956),
 (178.47826086956522, 43.47826086956522),
 (178.47826086956522, 43.91304347826087),
 (178.47826086956522, 44.34782608695652),
 (178.47826086956522, 44.78260869565217),
 (178.47826086956522, 45.21739130434783),
 (178.47826086956522, 45.65217391304348),
 (178.47826086956522, 46.086956521739125),
 (178.47826086956522, 46.52173913043478),
 (178.47826086956522, 46.95652173913044),
 (178.47826086956522, 47.391304347826086),
 (178.47826086956522, 47.826086956521735),
 (178.47826086956522, 48.26086956521739),
 (178.47826086956522, 48.69565217391305),
 (178.47826086956522, 49.130434782608695),
 (178.47826086956522, 49.565217391304344),
 (178.47826086956522, 50.0),
 (178.47826086956522, 50.434782608695656),
 (178.47826086956522, 50.869565217391305),
 (178.47826086956522, 51.30434782608695),
 (178.47826086956522, 51.73913043478261),
 (178.47826086956522, 52.173913043478265),
 (178.47826086956522, 52.608695652173914),
 (178.47826086956522, 53.04347826086956),
 (178.47826086956522, 53.47826086956522),
 (178.47826086956522, 53.91304347826087),
 (178.47826086956522, 54.347826086956516),
 (178.47826086956522, 54.78260869565217),
 (178.47826086956522, 55.21739130434783),
 (178.47826086956522, 55.65217391304348),
 (178.47826086956522, 56.086956521739125),
 (178.47826086956522, 56.52173913043478),
 (178.47826086956522, 56.95652173913044),
 (178.47826086956522, 57.391304347826086),
 (178.47826086956522, 57.826086956521735),
 (178.47826086956522, 58.26086956521739),
 (178.47826086956522, 58.69565217391305),
 (178.47826086956522, 59.130434782608695),
 (178.47826086956522, 59.565217391304344),
 (178.47826086956522, 60.0),
 (179.42028985507247, 30.0),
 (179.42028985507247, 30.434782608695652),
 (179.42028985507247, 30.869565217391305),
 (179.42028985507247, 31.304347826086957),
 (179.42028985507247, 31.73913043478261),
 (179.42028985507247, 32.17391304347826),
 (179.42028985507247, 32.608695652173914),
 (179.42028985507247, 33.04347826086956),
 (179.42028985507247, 33.47826086956522),
 (179.42028985507247, 33.91304347826087),
 (179.42028985507247, 34.34782608695652),
 (179.42028985507247, 34.78260869565217),
 (179.42028985507247, 35.21739130434783),
 (179.42028985507247, 35.65217391304348),
 (179.42028985507247, 36.08695652173913),
 (179.42028985507247, 36.52173913043478),
 (179.42028985507247, 36.95652173913044),
 (179.42028985507247, 37.391304347826086),
 (179.42028985507247, 37.82608695652174),
 (179.42028985507247, 38.26086956521739),
 (179.42028985507247, 38.69565217391305),
 (179.42028985507247, 39.130434782608695),
 (179.42028985507247, 39.565217391304344),
 (179.42028985507247, 40.0),
 (179.42028985507247, 40.434782608695656),
 (179.42028985507247, 40.869565217391305),
 (179.42028985507247, 41.30434782608695),
 (179.42028985507247, 41.73913043478261),
 (179.42028985507247, 42.17391304347826),
 (179.42028985507247, 42.608695652173914),
 (179.42028985507247, 43.04347826086956),
 (179.42028985507247, 43.47826086956522),
 (179.42028985507247, 43.91304347826087),
 (179.42028985507247, 44.34782608695652),
 (179.42028985507247, 44.78260869565217),
 (179.42028985507247, 45.21739130434783),
 (179.42028985507247, 45.65217391304348),
 (179.42028985507247, 46.086956521739125),
 (179.42028985507247, 46.52173913043478),
 (179.42028985507247, 46.95652173913044),
 (179.42028985507247, 47.391304347826086),
 (179.42028985507247, 47.826086956521735),
 (179.42028985507247, 48.26086956521739),
 (179.42028985507247, 48.69565217391305),
 (179.42028985507247, 49.130434782608695),
 (179.42028985507247, 49.565217391304344),
 (179.42028985507247, 50.0),
 (179.42028985507247, 50.434782608695656),
 (179.42028985507247, 50.869565217391305),
 (179.42028985507247, 51.30434782608695),
 (179.42028985507247, 51.73913043478261),
 (179.42028985507247, 52.173913043478265),
 (179.42028985507247, 52.608695652173914),
 (179.42028985507247, 53.04347826086956),
 (179.42028985507247, 53.47826086956522),
 (179.42028985507247, 53.91304347826087),
 (179.42028985507247, 54.347826086956516),
 (179.42028985507247, 54.78260869565217),
 (179.42028985507247, 55.21739130434783),
 (179.42028985507247, 55.65217391304348),
 (179.42028985507247, 56.086956521739125),
 (179.42028985507247, 56.52173913043478),
 (179.42028985507247, 56.95652173913044),
 (179.42028985507247, 57.391304347826086),
 (179.42028985507247, 57.826086956521735),
 (179.42028985507247, 58.26086956521739),
 (179.42028985507247, 58.69565217391305),
 (179.42028985507247, 59.130434782608695),
 (179.42028985507247, 59.565217391304344),
 (179.42028985507247, 60.0),
 (180.36231884057972, 30.0),
 (180.36231884057972, 30.434782608695652),
 (180.36231884057972, 30.869565217391305),
 (180.36231884057972, 31.304347826086957),
 (180.36231884057972, 31.73913043478261),
 (180.36231884057972, 32.17391304347826),
 (180.36231884057972, 32.608695652173914),
 (180.36231884057972, 33.04347826086956),
 (180.36231884057972, 33.47826086956522),
 (180.36231884057972, 33.91304347826087),
 (180.36231884057972, 34.34782608695652),
 (180.36231884057972, 34.78260869565217),
 (180.36231884057972, 35.21739130434783),
 (180.36231884057972, 35.65217391304348),
 (180.36231884057972, 36.08695652173913),
 (180.36231884057972, 36.52173913043478),
 (180.36231884057972, 36.95652173913044),
 (180.36231884057972, 37.391304347826086),
 (180.36231884057972, 37.82608695652174),
 (180.36231884057972, 38.26086956521739),
 (180.36231884057972, 38.69565217391305),
 (180.36231884057972, 39.130434782608695),
 (180.36231884057972, 39.565217391304344),
 (180.36231884057972, 40.0),
 (180.36231884057972, 40.434782608695656),
 (180.36231884057972, 40.869565217391305),
 (180.36231884057972, 41.30434782608695),
 (180.36231884057972, 41.73913043478261),
 (180.36231884057972, 42.17391304347826),
 (180.36231884057972, 42.608695652173914),
 (180.36231884057972, 43.04347826086956),
 (180.36231884057972, 43.47826086956522),
 (180.36231884057972, 43.91304347826087),
 (180.36231884057972, 44.34782608695652),
 (180.36231884057972, 44.78260869565217),
 (180.36231884057972, 45.21739130434783),
 (180.36231884057972, 45.65217391304348),
 (180.36231884057972, 46.086956521739125),
 (180.36231884057972, 46.52173913043478),
 (180.36231884057972, 46.95652173913044),
 (180.36231884057972, 47.391304347826086),
 (180.36231884057972, 47.826086956521735),
 (180.36231884057972, 48.26086956521739),
 (180.36231884057972, 48.69565217391305),
 (180.36231884057972, 49.130434782608695),
 (180.36231884057972, 49.565217391304344),
 (180.36231884057972, 50.0),
 (180.36231884057972, 50.434782608695656),
 (180.36231884057972, 50.869565217391305),
 (180.36231884057972, 51.30434782608695),
 (180.36231884057972, 51.73913043478261),
 (180.36231884057972, 52.173913043478265),
 (180.36231884057972, 52.608695652173914),
 (180.36231884057972, 53.04347826086956),
 (180.36231884057972, 53.47826086956522),
 (180.36231884057972, 53.91304347826087),
 (180.36231884057972, 54.347826086956516),
 (180.36231884057972, 54.78260869565217),
 (180.36231884057972, 55.21739130434783),
 (180.36231884057972, 55.65217391304348),
 (180.36231884057972, 56.086956521739125),
 (180.36231884057972, 56.52173913043478),
 (180.36231884057972, 56.95652173913044),
 (180.36231884057972, 57.391304347826086),
 (180.36231884057972, 57.826086956521735),
 (180.36231884057972, 58.26086956521739),
 (180.36231884057972, 58.69565217391305),
 (180.36231884057972, 59.130434782608695),
 (180.36231884057972, 59.565217391304344),
 (180.36231884057972, 60.0),
 (181.30434782608697, 30.0),
 (181.30434782608697, 30.434782608695652),
 (181.30434782608697, 30.869565217391305),
 (181.30434782608697, 31.304347826086957),
 (181.30434782608697, 31.73913043478261),
 (181.30434782608697, 32.17391304347826),
 (181.30434782608697, 32.608695652173914),
 (181.30434782608697, 33.04347826086956),
 (181.30434782608697, 33.47826086956522),
 (181.30434782608697, 33.91304347826087),
 (181.30434782608697, 34.34782608695652),
 (181.30434782608697, 34.78260869565217),
 (181.30434782608697, 35.21739130434783),
 (181.30434782608697, 35.65217391304348),
 (181.30434782608697, 36.08695652173913),
 (181.30434782608697, 36.52173913043478),
 (181.30434782608697, 36.95652173913044),
 (181.30434782608697, 37.391304347826086),
 (181.30434782608697, 37.82608695652174),
 (181.30434782608697, 38.26086956521739),
 (181.30434782608697, 38.69565217391305),
 (181.30434782608697, 39.130434782608695),
 (181.30434782608697, 39.565217391304344),
 (181.30434782608697, 40.0),
 (181.30434782608697, 40.434782608695656),
 (181.30434782608697, 40.869565217391305),
 (181.30434782608697, 41.30434782608695),
 (181.30434782608697, 41.73913043478261),
 (181.30434782608697, 42.17391304347826),
 (181.30434782608697, 42.608695652173914),
 (181.30434782608697, 43.04347826086956),
 (181.30434782608697, 43.47826086956522),
 (181.30434782608697, 43.91304347826087),
 (181.30434782608697, 44.34782608695652),
 (181.30434782608697, 44.78260869565217),
 (181.30434782608697, 45.21739130434783),
 (181.30434782608697, 45.65217391304348),
 (181.30434782608697, 46.086956521739125),
 (181.30434782608697, 46.52173913043478),
 (181.30434782608697, 46.95652173913044),
 (181.30434782608697, 47.391304347826086),
 (181.30434782608697, 47.826086956521735),
 (181.30434782608697, 48.26086956521739),
 (181.30434782608697, 48.69565217391305),
 (181.30434782608697, 49.130434782608695),
 (181.30434782608697, 49.565217391304344),
 (181.30434782608697, 50.0),
 (181.30434782608697, 50.434782608695656),
 (181.30434782608697, 50.869565217391305),
 (181.30434782608697, 51.30434782608695),
 (181.30434782608697, 51.73913043478261),
 (181.30434782608697, 52.173913043478265),
 (181.30434782608697, 52.608695652173914),
 (181.30434782608697, 53.04347826086956),
 (181.30434782608697, 53.47826086956522),
 (181.30434782608697, 53.91304347826087),
 (181.30434782608697, 54.347826086956516),
 (181.30434782608697, 54.78260869565217),
 (181.30434782608697, 55.21739130434783),
 (181.30434782608697, 55.65217391304348),
 (181.30434782608697, 56.086956521739125),
 (181.30434782608697, 56.52173913043478),
 (181.30434782608697, 56.95652173913044),
 (181.30434782608697, 57.391304347826086),
 (181.30434782608697, 57.826086956521735),
 (181.30434782608697, 58.26086956521739),
 (181.30434782608697, 58.69565217391305),
 (181.30434782608697, 59.130434782608695),
 (181.30434782608697, 59.565217391304344),
 (181.30434782608697, 60.0),
 (182.2463768115942, 30.0),
 (182.2463768115942, 30.434782608695652),
 (182.2463768115942, 30.869565217391305),
 (182.2463768115942, 31.304347826086957),
 (182.2463768115942, 31.73913043478261),
 (182.2463768115942, 32.17391304347826),
 (182.2463768115942, 32.608695652173914),
 (182.2463768115942, 33.04347826086956),
 (182.2463768115942, 33.47826086956522),
 (182.2463768115942, 33.91304347826087),
 (182.2463768115942, 34.34782608695652),
 (182.2463768115942, 34.78260869565217),
 (182.2463768115942, 35.21739130434783),
 (182.2463768115942, 35.65217391304348),
 (182.2463768115942, 36.08695652173913),
 (182.2463768115942, 36.52173913043478),
 (182.2463768115942, 36.95652173913044),
 (182.2463768115942, 37.391304347826086),
 (182.2463768115942, 37.82608695652174),
 (182.2463768115942, 38.26086956521739),
 (182.2463768115942, 38.69565217391305),
 (182.2463768115942, 39.130434782608695),
 (182.2463768115942, 39.565217391304344),
 (182.2463768115942, 40.0),
 (182.2463768115942, 40.434782608695656),
 (182.2463768115942, 40.869565217391305),
 (182.2463768115942, 41.30434782608695),
 (182.2463768115942, 41.73913043478261),
 (182.2463768115942, 42.17391304347826),
 (182.2463768115942, 42.608695652173914),
 (182.2463768115942, 43.04347826086956),
 (182.2463768115942, 43.47826086956522),
 (182.2463768115942, 43.91304347826087),
 (182.2463768115942, 44.34782608695652),
 (182.2463768115942, 44.78260869565217),
 (182.2463768115942, 45.21739130434783),
 (182.2463768115942, 45.65217391304348),
 (182.2463768115942, 46.086956521739125),
 (182.2463768115942, 46.52173913043478),
 (182.2463768115942, 46.95652173913044),
 (182.2463768115942, 47.391304347826086),
 (182.2463768115942, 47.826086956521735),
 (182.2463768115942, 48.26086956521739),
 (182.2463768115942, 48.69565217391305),
 (182.2463768115942, 49.130434782608695),
 (182.2463768115942, 49.565217391304344),
 (182.2463768115942, 50.0),
 (182.2463768115942, 50.434782608695656),
 (182.2463768115942, 50.869565217391305),
 (182.2463768115942, 51.30434782608695),
 (182.2463768115942, 51.73913043478261),
 (182.2463768115942, 52.173913043478265),
 (182.2463768115942, 52.608695652173914),
 (182.2463768115942, 53.04347826086956),
 (182.2463768115942, 53.47826086956522),
 (182.2463768115942, 53.91304347826087),
 (182.2463768115942, 54.347826086956516),
 (182.2463768115942, 54.78260869565217),
 (182.2463768115942, 55.21739130434783),
 (182.2463768115942, 55.65217391304348),
 (182.2463768115942, 56.086956521739125),
 (182.2463768115942, 56.52173913043478),
 (182.2463768115942, 56.95652173913044),
 (182.2463768115942, 57.391304347826086),
 (182.2463768115942, 57.826086956521735),
 (182.2463768115942, 58.26086956521739),
 (182.2463768115942, 58.69565217391305),
 (182.2463768115942, 59.130434782608695),
 (182.2463768115942, 59.565217391304344),
 (182.2463768115942, 60.0),
 (183.18840579710144, 30.0),
 (183.18840579710144, 30.434782608695652),
 (183.18840579710144, 30.869565217391305),
 (183.18840579710144, 31.304347826086957),
 (183.18840579710144, 31.73913043478261),
 (183.18840579710144, 32.17391304347826),
 (183.18840579710144, 32.608695652173914),
 (183.18840579710144, 33.04347826086956),
 (183.18840579710144, 33.47826086956522),
 (183.18840579710144, 33.91304347826087),
 (183.18840579710144, 34.34782608695652),
 (183.18840579710144, 34.78260869565217),
 (183.18840579710144, 35.21739130434783),
 (183.18840579710144, 35.65217391304348),
 (183.18840579710144, 36.08695652173913),
 (183.18840579710144, 36.52173913043478),
 (183.18840579710144, 36.95652173913044),
 (183.18840579710144, 37.391304347826086),
 (183.18840579710144, 37.82608695652174),
 (183.18840579710144, 38.26086956521739),
 ...]

There are 4900 tuples in this list (\(70 \cdot 70\)).

len(list(product(x,y)))

4900

We convert this into a DataFrame with 4900 rows and two columns.

df_art = pd.DataFrame(list(product(x,y)))

We give the columns the same names as our input features.

df_art.columns = cols

Here is what df_art looks like.

df_art

	flipper_length_mm	bill_length_mm
0	170.0	30.000000
1	170.0	30.434783
2	170.0	30.869565
3	170.0	31.304348
4	170.0	31.739130
...	...	...
4895	235.0	58.260870
4896	235.0	58.695652
4897	235.0	59.130435
4898	235.0	59.565217
4899	235.0	60.000000

4900 rows × 2 columns

We now add a prediction column. (A warning would be raised if we didn’t have the same column names as when we fit the classifier using clf.fit.)

df_art["pred"] = clf.predict(df_art[cols])

Here is the new DataFrame. For example, our classifier predicts a penguin with flipper length 170 and bill length 30 is an Adelie penguin. (Notice how there is probably no actual penguin with these measurements.)

df_art

	flipper_length_mm	bill_length_mm	pred
0	170.0	30.000000	Adelie
1	170.0	30.434783	Adelie
2	170.0	30.869565	Adelie
3	170.0	31.304348	Adelie
4	170.0	31.739130	Adelie
...	...	...	...
4895	235.0	58.260870	Gentoo
4896	235.0	58.695652	Gentoo
4897	235.0	59.130435	Gentoo
4898	235.0	59.565217	Gentoo
4899	235.0	60.000000	Gentoo

4900 rows × 3 columns

• Make another Altair scatter plot of the predicted species, this time using df_art.

The most important thing to recognize from the following picture is that there are three regions (corresponding to the three classes) and that regions are separated by linear boundaries. This partly explains why LogisticRegression is defined in the linear_model library of scikit-learn.

alt.Chart(df_art).mark_circle().encode(
    x=alt.X(cols[0], scale=alt.Scale(zero=False)),
    y=alt.Y(cols[1], scale=alt.Scale(zero=False)),
    color="pred"
)

We had some extra time, so let’s see how to use np.meshgrid instead of itertools.product. Here is a reminder of what x looks like.

x

array([170.        , 170.94202899, 171.88405797, 172.82608696,
       173.76811594, 174.71014493, 175.65217391, 176.5942029 ,
       177.53623188, 178.47826087, 179.42028986, 180.36231884,
       181.30434783, 182.24637681, 183.1884058 , 184.13043478,
       185.07246377, 186.01449275, 186.95652174, 187.89855072,
       188.84057971, 189.7826087 , 190.72463768, 191.66666667,
       192.60869565, 193.55072464, 194.49275362, 195.43478261,
       196.37681159, 197.31884058, 198.26086957, 199.20289855,
       200.14492754, 201.08695652, 202.02898551, 202.97101449,
       203.91304348, 204.85507246, 205.79710145, 206.73913043,
       207.68115942, 208.62318841, 209.56521739, 210.50724638,
       211.44927536, 212.39130435, 213.33333333, 214.27536232,
       215.2173913 , 216.15942029, 217.10144928, 218.04347826,
       218.98550725, 219.92753623, 220.86956522, 221.8115942 ,
       222.75362319, 223.69565217, 224.63768116, 225.57971014,
       226.52173913, 227.46376812, 228.4057971 , 229.34782609,
       230.28985507, 231.23188406, 232.17391304, 233.11594203,
       234.05797101, 235.        ])

The meshgrid function returns two NumPy arrays (when we provide two inputs), corresponding to the Cartesian product.

xx, yy = np.meshgrid(x,y)

Notice how the output xx is actually two-dimensional.

xx

array([[170.        , 170.94202899, 171.88405797, ..., 233.11594203,
        234.05797101, 235.        ],
       [170.        , 170.94202899, 171.88405797, ..., 233.11594203,
        234.05797101, 235.        ],
       [170.        , 170.94202899, 171.88405797, ..., 233.11594203,
        234.05797101, 235.        ],
       ...,
       [170.        , 170.94202899, 171.88405797, ..., 233.11594203,
        234.05797101, 235.        ],
       [170.        , 170.94202899, 171.88405797, ..., 233.11594203,
        234.05797101, 235.        ],
       [170.        , 170.94202899, 171.88405797, ..., 233.11594203,
        234.05797101, 235.        ]])

Imagine grouping together the xx and yy entries. At the top we would get (170, 30), then we would get (approximately) (170.9, 30), and so on.

yy

array([[30.        , 30.        , 30.        , ..., 30.        ,
        30.        , 30.        ],
       [30.43478261, 30.43478261, 30.43478261, ..., 30.43478261,
        30.43478261, 30.43478261],
       [30.86956522, 30.86956522, 30.86956522, ..., 30.86956522,
        30.86956522, 30.86956522],
       ...,
       [59.13043478, 59.13043478, 59.13043478, ..., 59.13043478,
        59.13043478, 59.13043478],
       [59.56521739, 59.56521739, 59.56521739, ..., 59.56521739,
        59.56521739, 59.56521739],
       [60.        , 60.        , 60.        , ..., 60.        ,
        60.        , 60.        ]])

We get an error here (to me the error message is not very helpful) because we are using two-dimensional NumPy arrays for the columns. (Remember that xx and yy were two-dimensional.)

df_art2 = pd.DataFrame(
    {
        cols[0]: xx,
        cols[1]: yy
    }
)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_106/2658753697.py in <module>
      2     {
      3         cols[0]: xx,
----> 4         cols[1]: yy
      5     }
      6 )

/shared-libs/python3.7/py/lib/python3.7/site-packages/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
    527 
    528         elif isinstance(data, dict):
--> 529             mgr = init_dict(data, index, columns, dtype=dtype)
    530         elif isinstance(data, ma.MaskedArray):
    531             import numpy.ma.mrecords as mrecords

/shared-libs/python3.7/py/lib/python3.7/site-packages/pandas/core/internals/construction.py in init_dict(data, index, columns, dtype)
    285             arr if not is_datetime64tz_dtype(arr) else arr.copy() for arr in arrays
    286         ]
--> 287     return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
    288 
    289 

/shared-libs/python3.7/py/lib/python3.7/site-packages/pandas/core/internals/construction.py in arrays_to_mgr(arrays, arr_names, index, columns, dtype, verify_integrity)
     78         # figure out the index, if necessary
     79         if index is None:
---> 80             index = extract_index(arrays)
     81         else:
     82             index = ensure_index(index)

/shared-libs/python3.7/py/lib/python3.7/site-packages/pandas/core/internals/construction.py in extract_index(data)
    389 
    390         if not indexes and not raw_lengths:
--> 391             raise ValueError("If using all scalar values, you must pass an index")
    392 
    393         if have_series:

ValueError: If using all scalar values, you must pass an index

The array xx is a 70-by-70 array.

xx.shape

(70, 70)

We can reshape it to a one-dimensional array by calling xx.reshape(-1). Think of -1 as like a wild card, which says to make it however long is necessary to include all the data.

xx.reshape(-1).shape

(4900,)

This code no longer raises an error.

df_art2 = pd.DataFrame(
    {
        cols[0]: xx.reshape(-1),
        cols[1]: yy.reshape(-1)
    }
)

We can add a “pred” column just like above.

df_art2["pred"] = clf.predict(df_art2[cols])

Here is another illustration of the decision boundary. I believe it should be the exact same image as above. (But notice we are using df_art2 instead of df_art.)

alt.Chart(df_art2).mark_circle().encode(
    x=alt.X(cols[0], scale=alt.Scale(zero=False)),
    y=alt.Y(cols[1], scale=alt.Scale(zero=False)),
    color="pred"
)