Skip to main content

Class 12 - Task planning & visualization

Date: October 5, 2026

⚠️Homework
Homework 4 due October 11, 2026
‼️Ending Skills
Last week for skills: 0
🆕New Skills
First week for skills: 3

Reference

For all work with tables, refer to the Tables page in the menu at the top of the page!

Skills: None​

Notes (review before class)​

Task Plans​

  • A task plan decomposes a problem into steps (sub-tasks) you know how to implement, with a built-in function or your own: a programming to-do list, as bullets, a diagram of the data's transformations, or both.
  • Task planning is what we did for the discount chart on Class 11: start from a concrete example, find a function in the docs, make the data work with it.

Strategy: Creating a Task Plan

  1. Make a concrete example of the desired output, for an input you pick: big enough to show its features, small enough to work by hand (4-6 table rows).
  2. Identify functions (known, or from the documentation) that might help get from input to output.
  3. Develop a sequence of steps that could solve the problem; in pictures, show the example's intermediate data and the operations between them.
  4. Repeat step 3 on sub-tasks until you could code each step.

The discount chart's plan, on paper:

 Name       Num Tickets   Discount Code
 "Jaime"    2             "BIRTHDAY"
 "Darren"   4             "birthday"
 "Nya"      3             " "
 "Luis"     5             "Student"
               |
               |  using transform_column
               v
 Name       Num Tickets   Discount Code
 "Jaime"    2             "BIRTHDAY"
 "Darren"   4             "BIRTHDAY"
 "Nya"      3             "none"
 "Luis"     5             "STUDENT"
               |
               |  using freq_bar_chart
               v
 bars: BIRTHDAY 2, none 1, STUDENT 1
  • To turn a plan into code, write each step, passing its output to the next.
  • A plan can also transform a table in different ways, then combine the results, e.g., the difference between the largest and lowest midterm scores:
     name       midterm   final
     "Rania"    86        82
     "Bernd"    82        90
     "Annika"   93        85
     "Cole"     78        89
         |                     |
         |  sort, look up      |  sort, look up
         v                     v
         93                    78
           \                  /
            '--->  93 - 78  <'
  • Try: plan how to find the student with the largest improvement from midterm to final, given a table of names and both exams' scores.
  • If you aren't sure how to approach a problem, don't start by writing code. Plan until you understand the problem. Coding first only seems faster.

Mixed Types​

A column in a table(...) may mix types (45 and "45"); a function normalizing it must accept both:

  • isinstance(v, str) is True if v is a string.
  • The annotation float | str means "a float or a str" (| means "or").
  • Convert the strings with a helper like string_to_int (Class 11), using float for decimals.

Binning​

Clean data may still need preparing: charting small, medium, and large orders needs a scale label for each order.

  • Binning reduces values (like numbers of tickets) to a few categories, the bins (small/medium/large; morning/afternoon/evening).
  • Write a function that computes a value's bin:
def order_scale_label(num_tickets: int) -> str:
    """categorizes a number of tickets as small, medium, or large"""
    if num_tickets >= 10:
        return "large"
    elif num_tickets >= 5:
        return "medium"
    else:
        return "small"

def test_order_scale_label():
    assert order_scale_label(2) == "small"
    assert order_scale_label(5) == "medium"
    assert order_scale_label(9) == "medium"
    assert order_scale_label(10) == "large"
  • Then add a column of bins with it (order_bin_data, below). It takes a number, not a row, so it's easy to test; a lambda passes it each row's value.

Splitting Columns​

  • s.split(sep) splits s around sep into a list of the pieces (lists: Class 13): "Josie Zhao".split(" ") is ["Josie", "Zhao"].
  • Get a piece by its position, from 0, in square brackets: "Josie Zhao".split(" ")[1] is "Zhao". len works on lists, too.
  • To sort by last name, we might replace Name with first-name and last-name columns (add_column, then select_columns to drop Name). But what about "Zander"? Middle names? Both parents' last names? Family name first? Non-Latin alphabets?
  • Names are context- and culture-dependent. Excluding names outside your population is okay; learning later that you excluded data you need is a headache. So examine your table structure for its assumptions about your data, and choose it knowing whom it must represent. (Search "falsehoods programmers believe about names".)

Managing and Naming Tables​

  • Keep the original file and the corrected copy (to see people's errors, or try other fixes), and keep cleaned data separate from loaded data (table operations make new tables).
  • Name the loaded table, the cleaned table, and significant variations, not every intermediate table (consistent names reduce confusion between projects):
event_data_raw = load_table("https://raw.githubusercontent.com/neu-pdi/cs2000-public-resources/refs/heads/main/static/support/dcic-events-data.csv")

cleaned_event_data = (
    event_data_raw.transform_column("Num Tickets", int)
    .transform_column("Discount Code", cell_to_discount_code)
    .transform_column("Delivery", yes_to_email)
)

order_bin_data = cleaned_event_data.add_column("order-scale", lambda r: order_scale_label(r["Num Tickets"]))
  • transform_column produces a table, so we can call it on its own result; the parentheses let each step have its own line. (cell_to_discount_code, yes_to_email: Class 11.)
  • For a huge dataset, save the cleaned table to a file (Save CSV, on a table in the interactions panel), and load only that.

Visualizations and Plots​

The right plot depends on the question, and the kinds of variables (columns):

  • Quantitative: numeric, ordered with a consistent interval between values, meaningful to compute with.
  • Categorical: a fixed set of values, with no meaningful computation beyond (perhaps) ordering; usually characteristics of samples.
  • Last names, course grades, zip codes: which kind is each?

Plots (table methods; see Tables):

  • scatter_plot: relates two quantitative variables.
  • freq_bar_chart: how often each categorical value appears.
  • histogram: how quantitative values fall into equal-size intervals.
  • pie_chart: each categorical value's share of a total.

Try: which chart shows the most used discount code? Whether tickets per order relate to purchase time? Orders per delivery option? For the last:

cleaned_event_data.freq_bar_chart("Delivery")
  • Bars: email (4), pickup (2), and emall (1), Alvina's typo.
  • Plots also sanity-check data: never trust a dataset without first making sure its values make sense. For datasets too big to inspect by hand, plots and summaries find errors.

Summary: Managing a Data Analysis​

  1. Think about each column's plausible values and likely errors, given how the data were collected.
  2. Check for errors (look, plot, filter), then correct or normalize, at the source (if you control it) or in code.
  3. Store the cleaned table (a name, or a file), leaving the raw data intact.
  4. Prepare the data for each question (new columns, bins, combined tables); name one final table, or one per question.
  5. Analyze, with statistics, plots, and interpretations suited to the question and variables; note which file and code produced each result.

Beyond this course: machine learning predicts from datasets with programs others wrote, and needs these skills to be used effectively and responsibly. Its issues: Are the samples representative? Do correlations lead to discrimination (the hiring case, Class 6)? Is data collected for one purpose misused? How are results interpreted?

Recap (25 mins)​

We have a table of weather data, and want a chart of the number of cold (below 40F), mild (40F up to 60F), and hot (60F and up) days. Read the plan, then the program that follows it:

  1. Normalize the temperature column, so it's all numbers. (precipitation has a problem too, a negative number, but our task doesn't use it.)
  2. Add a column with each day's bin: "cold", "mild", or "hot".
  3. Chart the new column.
weather_data = table(
    ["date", "temperature", "precipitation"],
    [
        ["2025-01-01", 62, 0.1],
        ["2025-01-02", "45", 3],
        ["2025-01-03", 28, 0.2],
        ["2025-01-04", 55, -1],
        ["2025-01-05", 90, 0],
    ],
)

def string_to_number(s: str, default: float) -> float:
    """converts s to a number, or produces default if s does not represent a number"""
    try:
        return float(s)
    except ValueError:
        return default

def normalize_temp(v: float | str) -> float:
    """given a number or a string that represents a number, converts to a number"""
    if isinstance(v, str):
        return string_to_number(v, 0)
    else:
        return v

def test_normalize_temp():
    assert normalize_temp(10) == 10
    assert normalize_temp("13") == 13

def bucket_temp(t: float) -> str:
    """numbers < 40 turn into 'cold', >=40 and < 60 to 'mild' and >=60 to 'hot'"""
    if t < 40:
        return "cold"
    elif t < 60:
        return "mild"
    else:
        return "hot"

def test_bucket_temp():
    assert bucket_temp(-10) == "cold"
    assert bucket_temp(0) == "cold"
    assert bucket_temp(39.9) == "cold"
    assert bucket_temp(40) == "mild"
    assert bucket_temp(58) == "mild"
    assert bucket_temp(60) == "hot"
    assert bucket_temp(100) == "hot"

fixed_data = weather_data.transform_column("temperature", normalize_temp)
with_buckets = fixed_data.add_column("temp-category", lambda r: bucket_temp(r["temperature"]))
with_buckets.freq_bar_chart("temp-category")
  • How tall is each bar in the chart? Which line carries out each step of the plan?
  • Why does test_bucket_temp check 39.9, 40, and 60?
  • If the second day's temperature were "forty-five", which bar would that day be counted in? Is that right?

Class Activity (30 mins)​

  1. Using the same data, visualize the number of days that were "dry" (no rain), "drizzly" (< 1" of rain), and "wet" (>= 1" of rain). It's up to you to decide how to handle the bad data! Think about what it might represent! First, make a plan!
  2. Consider you have the following table:
employees = table(
    ["full-name", "department"],
    [
        ["Jordan Smith", "Sales"],
        ["Alexandra Lee", "Engineering"],
        ["Sam", "Marketing"],
        ["Ng, Alice", "Operations"],
    ],
)

You want to have in your data, in addition to "full name", columns for given name (or first name) and family name (or last name). Make a plan for computing these two columns, thinking carefully about what different cases you might have to consider. Are all of the cases you thought of reflected in the example above? If not, add more examples to the data.

Wrap up (5 mins)​

  • Planning is useful for all programming, and all good programmers do it -- probably, as you gain experience, simpler things you will plan only in your head, and more complex things you will plan on paper (or on whiteboards, etc). But the process of planning remains the same, whatever the scale!