What is data quality?

IBM’s definition of data quality:

“Data quality measures how well a dataset meets criteria for accuracy, completeness, validity, consistency, uniqueness, timeliness, and fitness for purpose”

Aspects of data quality?

• Accuracy: Do the data reflect reality / truth?
• Completeness: Are the data usable or complete (no missing people, values, etc. beyond random)
  
• Uniqueness: There is no duplicated data
• Validity: Do the data have the correct properties (values, ranges, etc.)
• Consistency: When integrating across multiple data sources, information should converge across sources and match reality
• Timeliness: Can the data be maintained and distributed within a specified time frame
• Fitness for purpose: Do the data meet your research need?

Why should we care about data quality?

• You aren’t responsible for poor quality data you receive, but you are responsible for the data products you work with – that is, you are responsible for improving data quality
• Poor quality data threatens scientific integrity
  
• Poor quality data are a pain for you to work with and for others to work with

What can data quality do for my career?

• The virtuous cycle of data cleaning
  • Some people get a reputation for getting their data, analyses, etc. right
  • This is important for publications, grant funding, etc.
  • It tends to be inter-generational – you inherit some of your reputation on this from your advisor
  • Start paying it forward now to build your own career, whether it’s in academia or industry

How do I ensure data quality?

The Towards Data Science website has a nice definition of EDA:

“Exploratory Data Analysis refers to the critical process of performing initial investigations on data so as to discover patterns,to spot anomalies,to test hypothesis and to check assumptions with the help of summary statistics”

• So EDA is basically a fancy word for the descriptive statistics you’ve been learning about for years

How do I ensure data quality?

I think about “exploratory data analysis for data quality”

• Investigating values and patterns of variables from “input data”
• Identifying and cleaning errors or values that need to be changed
• Creating analysis variables
• Checking values of analysis variables against values of input variables

How I will teach exploratory data analysis

Will teach exploratory data analysis (EDA) in two sub-sections:

1. Provide “Guidelines for EDA”
   • Less about coding, more about practices you should follow and mentality necessary to ensure high data quality
2. Introduce “Tools of EDA”:
   • Demonstrate code to investigate variables and relationship between variables
   • Most of these tools are just the application of programming skills you have already learned (or will learn soon!)

Guidelines for “EDA for data quality”

Assume that your goal in “EDA for data quality” is to investigate “input” data sources and create “analysis variables”

• Usually, your analysis dataset will incorporate multiple sources of input data, including data you collect (primary data) and/or data collected by others (secondary data)

Guidelines for “EDA for data quality”

EDA is not a linear process, and the process will vary across people and projects Some broad steps:

1. Understand how input data sources were created
   • e.g., when working with survey data, have survey questionnaire and codebooks on hand (watch out for skip patterns!!!)
2. For each input data source, identify the “unit of analysis” and which combination of variables uniquely identify observations
3. Investigate patterns in input variables
4. Create analysis variable from input variable(s)
5. Verify that analysis variable is created correctly through descriptive statistics that compare values of input variable(s) against values of the analysis variable

Guidelines for “EDA for data quality”

• It is critically important to step through EDA processes at multiple points during data cleaning, from the input / raw data to the output / analysis / clean data.

• Always be aware of missing values

• They will not always be coded as NA in input variables (e.g., some projects code them as 99, 99999, negative values, etc.)

“Unit of analysis” and which variables uniquely identify observations

“Unit of analysis” refers to “what does each observation represent” in an input data source

• If each obs represents a trial in an experiment, you have “trial level data”
• If each obs represents a participant, you have “participant level data”
• If each obs represents a sample, you have “sample-level data”
• If each obs represents a year, you have “year level data” (i.e. longitudinal)

“Unit of analysis” and which variables uniquely identify observations

How to identify unit of analysis

• data documentation

• investigating the data set

• This is very important because we often conduct analyses that span multiple units of analysis (e.g., between- v within-person, person- v stimuli-level, etc.)

• We have to be careful and thoughtful about identifiers that let us do that (important for joining data together, which will be the focus on our R workshop today)

Data we will use

Use read_csv() function from readr (loaded with tidyverse) to import .csv dataset into R.

library(plyr)
library(tidyverse)
soep_long <- read_csv(file="https://github.com/emoriebeck/psc290-data-FQ23/raw/main/04-workshops/03-week3-tidyr/gsoep.csv")
soep_long

# A tibble: 136,627 × 30
   Procedural__SID Procedural__household Demographic__DOB Demographic__Sex  year
             <dbl>                 <dbl>            <dbl>            <dbl> <dbl>
 1             901                    94             1951                2  2005
 2             901                    94             1951                2  2006
 3             901                    94             1951                2  2007
 4             901                    94             1951                2  2008
 5             901                    94             1951                2  2009
 6             901                    94             1951                2  2010
 7             901                    94             1951                2  2011
 8             901                    94             1951                2  2012
 9             901                    94             1951                2  2013
10             901                    94             1951                2  2014
# ℹ 136,617 more rows
# ℹ 25 more variables: Big5__C_thorough <dbl>, Big5__E_communic <dbl>,
#   Big5__A_coarse <dbl>, Big5__O_original <dbl>, Big5__N_worry <dbl>,
#   Big5__A_forgive <dbl>, Big5__C_lazy <dbl>, Big5__E_sociable <dbl>,
#   Big5__O_artistic <dbl>, Big5__N_nervous <dbl>, Big5__C_efficient <dbl>,
#   Big5__E_reserved <dbl>, Big5__A_friendly <dbl>, Big5__O_imagin <dbl>,
#   Big5__N_dealStress <dbl>, LifeEvent__ChldBrth <dbl>, …

Data we will use

Let’s examine the data [you must run this code chunk]

soep_long %>% names()
soep_long %>% names() %>% str()

str(soep_long) 

str(soep_long$LifeEvent__Married)
attributes(soep_long$LifeEvent__Married)
typeof(soep_long$LifeEvent__Married)
class(soep_long$LifeEvent__Married)

Rule 1

1. Never modify “input variable”; instead create new variable based on input variable(s)

• Always keep input variables used to create new variables

• I already did this before the data were loaded in. I renamed all the input variables with interpretable names and reshaped them so the time variable (year) is long and the other variables are wide

Rule 3

3. Developing a plan for creation of analysis variable

• e.g., for each possible value of input variables, what should value of analysis variable be?

• I do this in my codebooks, and this topic warrants a discussion in itself. This is our focal topic for next week!

• In this case, we want Big Five (EACNO) composites for each wave and to create composites of life events experienced across all years

Rule 4

soep_long <- soep_long %>%
  mutate_at(
    vars(contains("Big5"))
    , ~ifelse(. < 0 | is.na(.), NA, .)
    ) %>%
  mutate_at(
    vars(contains("LifeEvent"))
    , ~mapvalues(., seq(-7,1), c(rep(NA, 5), 0, NA, NA, 1), warn_missing = F)
    )

Rule 4

4. Write code to create analysis variable

rev_code <- c("Big5__A_coarse", "Big5__C_lazy", "Big5__E_reserved", "Big5__N_dealStress")
soep_long <- soep_long %>%
  mutate_at(
    vars(all_of(rev_code))
    , ~as.numeric(reverse.code(., keys = -1, mini = 1, maxi = 7))
    )

Rule 4

soep_2005 <- soep_long %>% filter(year == 2005) %>% select(-year)
cor.plot(soep_2005, diag = F)

Rule 5

5. Run descriptive checks to verify new variables are constructed correctly

Can “comment out” these checks, but don’t delete them

• Uh oh, Inf values popping up what went wrong?
• -Inf pops up when there were no non-missing values and you use na.rm = T
• Let’s recode those as NA

Rule 5

5. Run descriptive checks to verify new variables are constructed correctly

Can “comment out” these checks, but don’t delete them

soep_long <- soep_long %>%
  mutate_all(~ifelse(is.infinite(.) | is.nan(.), NA, .))

Rule 5

soep_long %>%
  filter(year == 2005) %>%
  select(Big5__E:LifeEvent__SepPart_ever) %>%
  cor.plot(., diag = F)

Rule 6

6. Document new variables with notes and labels

Again, I do this in my codebooks, so more on this next week!!

EDA

• One-way descriptive analyses (i.e,. focus on one variable)
  • Descriptive analyses for continuous variables
  • Descriptive analyses for discreet/categorical variables
• Two-way descriptive analyses (relationship between two variables)
  • Categorical by categorical
  • Categorical by continuous
  • Continuous by continuous
• Realistically, we’ve actually already covered all this above, so we’ll loop back to this after learning tidyr

Outline

Outline

tidyr

• Now, let’s build off what we learned from dplyr and focus on reshaping and merging our data.
• First, the reshapers:

1. pivot_longer(), which takes a “wide” format data frame and makes it long.
   
2. pivot_wider(), which takes a “long” format data frame and makes it wide.

tidyr

• Next, the mergers:

3. full_join(), which merges all rows in either data frame
   
4. inner_join(), which merges rows whose keys are present in both data frames
   
5. left_join(), which “prioritizes” the first data set
   
6. right_join(), which “prioritizes” the second data set

1. pivot_longer(): Basic Application

Let’s start with an easy one – one key, one value:

bfi %>%
  rownames_to_column("SID") %>%
  pivot_longer(
    cols = A1:O5
    , names_to = "item"
    , values_to = "values"
    , values_drop_na = T
  ) %>%
  print(n = 8)

# A tibble: 69,492 × 6
  SID   gender education   age item  values
  <chr>  <int>     <int> <int> <chr>  <int>
1 61617      1        NA    16 A1         2
2 61617      1        NA    16 A2         4
3 61617      1        NA    16 A3         3
4 61617      1        NA    16 A4         4
5 61617      1        NA    16 A5         4
6 61617      1        NA    16 C1         2
7 61617      1        NA    16 C2         3
8 61617      1        NA    16 C3         3
# ℹ 69,484 more rows

1. pivot_longer(): More Advanced Application

Now a harder one – two keys, one value:

bfi %>%
  rownames_to_column("SID") %>%
  pivot_longer(
    cols = A1:O5
    , names_to = c("trait", "item_num")
    , names_sep = -1
    , values_to = "values"
    , values_drop_na = T
  ) %>%
  print(n = 8)

# A tibble: 69,492 × 7
  SID   gender education   age trait item_num values
  <chr>  <int>     <int> <int> <chr> <chr>     <int>
1 61617      1        NA    16 A     1             2
2 61617      1        NA    16 A     2             4
3 61617      1        NA    16 A     3             3
4 61617      1        NA    16 A     4             4
5 61617      1        NA    16 A     5             4
6 61617      1        NA    16 C     1             2
7 61617      1        NA    16 C     2             3
8 61617      1        NA    16 C     3             3
# ℹ 69,484 more rows

2. pivot_wider(): Basic Application

bfi_long <- bfi %>%
  rownames_to_column("SID") %>%
  pivot_longer(
    cols = A1:O5
    , names_to = "item"
    , values_to = "values"
    , values_drop_na = T
  )

bfi_long %>%
  pivot_wider(
    names_from = "item"
    , values_from = "values"
  )

# A tibble: 2,800 × 29
   SID   gender education   age    A1    A2    A3    A4    A5    C1    C2    C3
   <chr>  <int>     <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
 1 61617      1        NA    16     2     4     3     4     4     2     3     3
 2 61618      2        NA    18     2     4     5     2     5     5     4     4
 3 61620      2        NA    17     5     4     5     4     4     4     5     4
 4 61621      2        NA    17     4     4     6     5     5     4     4     3
 5 61622      1        NA    17     2     3     3     4     5     4     4     5
 6 61623      2         3    21     6     6     5     6     5     6     6     6
 7 61624      1        NA    18     2     5     5     3     5     5     4     4
 8 61629      1         2    19     4     3     1     5     1     3     2     4
 9 61630      1         1    19     4     3     6     3     3     6     6     3
10 61633      2        NA    17     2     5     6     6     5     6     5     6
# ℹ 2,790 more rows
# ℹ 17 more variables: C4 <int>, C5 <int>, E1 <int>, E2 <int>, E3 <int>,
#   E4 <int>, E5 <int>, N1 <int>, N2 <int>, N3 <int>, N4 <int>, N5 <int>,
#   O1 <int>, O2 <int>, O3 <int>, O4 <int>, O5 <int>

2. pivot_wider(): More Advanced

bfi_long <- bfi %>%
  rownames_to_column("SID") %>%
  pivot_longer(
    cols = A1:O5
    , names_to = c("trait", "item_num")
    , names_sep = -1
    , values_to = "values"
    , values_drop_na = T
  )

bfi_long %>%
  pivot_wider(
    names_from = c("trait", "item_num")
    , values_from = "values"
    , names_sep = "_"
  )

# A tibble: 2,800 × 29
   SID   gender education   age   A_1   A_2   A_3   A_4   A_5   C_1   C_2   C_3
   <chr>  <int>     <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
 1 61617      1        NA    16     2     4     3     4     4     2     3     3
 2 61618      2        NA    18     2     4     5     2     5     5     4     4
 3 61620      2        NA    17     5     4     5     4     4     4     5     4
 4 61621      2        NA    17     4     4     6     5     5     4     4     3
 5 61622      1        NA    17     2     3     3     4     5     4     4     5
 6 61623      2         3    21     6     6     5     6     5     6     6     6
 7 61624      1        NA    18     2     5     5     3     5     5     4     4
 8 61629      1         2    19     4     3     1     5     1     3     2     4
 9 61630      1         1    19     4     3     6     3     3     6     6     3
10 61633      2        NA    17     2     5     6     6     5     6     5     6
# ℹ 2,790 more rows
# ℹ 17 more variables: C_4 <int>, C_5 <int>, E_1 <int>, E_2 <int>, E_3 <int>,
#   E_4 <int>, E_5 <int>, N_1 <int>, N_2 <int>, N_3 <int>, N_4 <int>,
#   N_5 <int>, O_1 <int>, O_2 <int>, O_3 <int>, O_4 <int>, O_5 <int>

2. pivot_wider(): A Little More Advanced

bfi_long %>%
  select(-item_num) %>%
  pivot_wider(
    names_from = "trait"
    , values_from = "values"
    , names_sep = "_"
    , values_fn = mean
  )

# A tibble: 2,800 × 9
   SID   gender education   age     A     C     E     N     O
   <chr>  <int>     <int> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
 1 61617      1        NA    16   3.4   3.2  3.4    2.8   3.8
 2 61618      2        NA    18   3.6   4    3      3.8   3.2
 3 61620      2        NA    17   4.4   4    3.8    3.6   3.6
 4 61621      2        NA    17   4.8   4.2  4      2.8   3.6
 5 61622      1        NA    17   3.4   3.6  3.6    3.2   3.2
 6 61623      2         3    21   5.6   4.4  4      3     3.8
 7 61624      1        NA    18   4     3.6  4.2    1.4   3.8
 8 61629      1         2    19   2.8   3    3.2    4.2   3.4
 9 61630      1         1    19   3.8   4.8  3.75   3.6   5  
10 61633      2        NA    17   4.8   4    3.6    4.2   3.6
# ℹ 2,790 more rows

The _join() Functions

The _join() Functions

• Let’s separate demographic and BFI data

bfi_only <- bfi %>% 
  rownames_to_column("SID") %>%
  select(SID, matches("[0-9]"))
bfi_only %>% as_tibble() %>% print(n = 6)

# A tibble: 2,800 × 26
  SID      A1    A2    A3    A4    A5    C1    C2    C3    C4    C5    E1    E2
  <chr> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 61617     2     4     3     4     4     2     3     3     4     4     3     3
2 61618     2     4     5     2     5     5     4     4     3     4     1     1
3 61620     5     4     5     4     4     4     5     4     2     5     2     4
4 61621     4     4     6     5     5     4     4     3     5     5     5     3
5 61622     2     3     3     4     5     4     4     5     3     2     2     2
6 61623     6     6     5     6     5     6     6     6     1     3     2     1
# ℹ 2,794 more rows
# ℹ 13 more variables: E3 <int>, E4 <int>, E5 <int>, N1 <int>, N2 <int>,
#   N3 <int>, N4 <int>, N5 <int>, O1 <int>, O2 <int>, O3 <int>, O4 <int>,
#   O5 <int>

bfi_dem <- bfi %>%
  rownames_to_column("SID") %>%
  select(SID, education, gender, age)
bfi_dem %>% as_tibble() %>% print(n = 6)

# A tibble: 2,800 × 4
  SID   education gender   age
  <chr>     <int>  <int> <int>
1 61617        NA      1    16
2 61618        NA      2    18
3 61620        NA      2    17
4 61621        NA      2    17
5 61622        NA      1    17
6 61623         3      2    21
# ℹ 2,794 more rows

3. full_join()

Most simply, we can put those back together keeping all observations.

bfi_only %>%
  full_join(bfi_dem) %>%
  as_tibble() %>%
  print(n = 6)

# A tibble: 2,800 × 29
  SID      A1    A2    A3    A4    A5    C1    C2    C3    C4    C5    E1    E2
  <chr> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 61617     2     4     3     4     4     2     3     3     4     4     3     3
2 61618     2     4     5     2     5     5     4     4     3     4     1     1
3 61620     5     4     5     4     4     4     5     4     2     5     2     4
4 61621     4     4     6     5     5     4     4     3     5     5     5     3
5 61622     2     3     3     4     5     4     4     5     3     2     2     2
6 61623     6     6     5     6     5     6     6     6     1     3     2     1
# ℹ 2,794 more rows
# ℹ 16 more variables: E3 <int>, E4 <int>, E5 <int>, N1 <int>, N2 <int>,
#   N3 <int>, N4 <int>, N5 <int>, O1 <int>, O2 <int>, O3 <int>, O4 <int>,
#   O5 <int>, education <int>, gender <int>, age <int>

bfi %>%
  rownames_to_column("SID") %>%
  as_tibble() %>%
  print(n = 6)

# A tibble: 2,800 × 29
  SID      A1    A2    A3    A4    A5    C1    C2    C3    C4    C5    E1    E2
  <chr> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 61617     2     4     3     4     4     2     3     3     4     4     3     3
2 61618     2     4     5     2     5     5     4     4     3     4     1     1
3 61620     5     4     5     4     4     4     5     4     2     5     2     4
4 61621     4     4     6     5     5     4     4     3     5     5     5     3
5 61622     2     3     3     4     5     4     4     5     3     2     2     2
6 61623     6     6     5     6     5     6     6     6     1     3     2     1
# ℹ 2,794 more rows
# ℹ 16 more variables: E3 <int>, E4 <int>, E5 <int>, N1 <int>, N2 <int>,
#   N3 <int>, N4 <int>, N5 <int>, O1 <int>, O2 <int>, O3 <int>, O4 <int>,
#   O5 <int>, gender <int>, education <int>, age <int>

4. inner_join()

We can also keep all rows present in both data frames

bfi_dem %>%
  filter(row_number() %in% 1:1700) %>%
  inner_join(
    bfi_only %>%
      filter(row_number() %in% 1200:2800)
  ) %>%
  as_tibble() %>%
  print(n = 6)

# A tibble: 501 × 29
  SID   education gender   age    A1    A2    A3    A4    A5    C1    C2    C3
  <chr>     <int>  <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 64151         3      2    18     1     5     6     5     5     5     6     5
2 64152         4      2    29     1     5     6     5     5     2     1     4
3 64154         5      1    46     2     5     6     5     6     6     6     6
4 64155         5      1    58     5     4     4     4     5     4     4     5
5 64156         5      2    38     1     4     6     6     6     4     4     5
6 64158         5      2    27     2     3     1     1     1     4     2     2
# ℹ 495 more rows
# ℹ 17 more variables: C4 <int>, C5 <int>, E1 <int>, E2 <int>, E3 <int>,
#   E4 <int>, E5 <int>, N1 <int>, N2 <int>, N3 <int>, N4 <int>, N5 <int>,
#   O1 <int>, O2 <int>, O3 <int>, O4 <int>, O5 <int>

bfi %>%
  rownames_to_column("SID") %>%
  as_tibble() %>%
  print(n = 6)

# A tibble: 2,800 × 29
  SID      A1    A2    A3    A4    A5    C1    C2    C3    C4    C5    E1    E2
  <chr> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 61617     2     4     3     4     4     2     3     3     4     4     3     3
2 61618     2     4     5     2     5     5     4     4     3     4     1     1
3 61620     5     4     5     4     4     4     5     4     2     5     2     4
4 61621     4     4     6     5     5     4     4     3     5     5     5     3
5 61622     2     3     3     4     5     4     4     5     3     2     2     2
6 61623     6     6     5     6     5     6     6     6     1     3     2     1
# ℹ 2,794 more rows
# ℹ 16 more variables: E3 <int>, E4 <int>, E5 <int>, N1 <int>, N2 <int>,
#   N3 <int>, N4 <int>, N5 <int>, O1 <int>, O2 <int>, O3 <int>, O4 <int>,
#   O5 <int>, gender <int>, education <int>, age <int>

5. left_join()

Or all rows present in the left (first) data frame, perhaps if it’s a subset of people with complete data

bfi_dem %>%
  drop_na() %>%
  left_join(bfi_only) %>%
  as_tibble() %>%
  print(n = 6)

# A tibble: 2,577 × 29
  SID   education gender   age    A1    A2    A3    A4    A5    C1    C2    C3
  <chr>     <int>  <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 61623         3      2    21     6     6     5     6     5     6     6     6
2 61629         2      1    19     4     3     1     5     1     3     2     4
3 61630         1      1    19     4     3     6     3     3     6     6     3
4 61634         1      1    21     4     4     5     6     5     4     3     5
5 61640         1      1    17     4     5     2     2     1     5     5     5
6 61661         5      1    68     1     5     6     5     6     4     3     2
# ℹ 2,571 more rows
# ℹ 17 more variables: C4 <int>, C5 <int>, E1 <int>, E2 <int>, E3 <int>,
#   E4 <int>, E5 <int>, N1 <int>, N2 <int>, N3 <int>, N4 <int>, N5 <int>,
#   O1 <int>, O2 <int>, O3 <int>, O4 <int>, O5 <int>

bfi %>%
  rownames_to_column("SID") %>%
  as_tibble() %>%
  print(n = 6)

# A tibble: 2,800 × 29
  SID      A1    A2    A3    A4    A5    C1    C2    C3    C4    C5    E1    E2
  <chr> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 61617     2     4     3     4     4     2     3     3     4     4     3     3
2 61618     2     4     5     2     5     5     4     4     3     4     1     1
3 61620     5     4     5     4     4     4     5     4     2     5     2     4
4 61621     4     4     6     5     5     4     4     3     5     5     5     3
5 61622     2     3     3     4     5     4     4     5     3     2     2     2
6 61623     6     6     5     6     5     6     6     6     1     3     2     1
# ℹ 2,794 more rows
# ℹ 16 more variables: E3 <int>, E4 <int>, E5 <int>, N1 <int>, N2 <int>,
#   N3 <int>, N4 <int>, N5 <int>, O1 <int>, O2 <int>, O3 <int>, O4 <int>,
#   O5 <int>, gender <int>, education <int>, age <int>

6. right_join()

Or all rows present in the right (second) data frame, such as I do when I join a codebook with raw data

bfi_dem %>%
  drop_na() %>%
  right_join(bfi_only) %>%
  as_tibble() %>%
  print(n = 6)

# A tibble: 2,800 × 29
  SID   education gender   age    A1    A2    A3    A4    A5    C1    C2    C3
  <chr>     <int>  <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 61623         3      2    21     6     6     5     6     5     6     6     6
2 61629         2      1    19     4     3     1     5     1     3     2     4
3 61630         1      1    19     4     3     6     3     3     6     6     3
4 61634         1      1    21     4     4     5     6     5     4     3     5
5 61640         1      1    17     4     5     2     2     1     5     5     5
6 61661         5      1    68     1     5     6     5     6     4     3     2
# ℹ 2,794 more rows
# ℹ 17 more variables: C4 <int>, C5 <int>, E1 <int>, E2 <int>, E3 <int>,
#   E4 <int>, E5 <int>, N1 <int>, N2 <int>, N3 <int>, N4 <int>, N5 <int>,
#   O1 <int>, O2 <int>, O3 <int>, O4 <int>, O5 <int>

bfi %>%
  rownames_to_column("SID") %>%
  as_tibble() %>%
  print(n = 6)

# A tibble: 2,800 × 29
  SID      A1    A2    A3    A4    A5    C1    C2    C3    C4    C5    E1    E2
  <chr> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 61617     2     4     3     4     4     2     3     3     4     4     3     3
2 61618     2     4     5     2     5     5     4     4     3     4     1     1
3 61620     5     4     5     4     4     4     5     4     2     5     2     4
4 61621     4     4     6     5     5     4     4     3     5     5     5     3
5 61622     2     3     3     4     5     4     4     5     3     2     2     2
6 61623     6     6     5     6     5     6     6     6     1     3     2     1
# ℹ 2,794 more rows
# ℹ 16 more variables: E3 <int>, E4 <int>, E5 <int>, N1 <int>, N2 <int>,
#   N3 <int>, N4 <int>, N5 <int>, O1 <int>, O2 <int>, O3 <int>, O4 <int>,
#   O5 <int>, gender <int>, education <int>, age <int>

EDA

• One-way descriptive analyses (i.e,. focus on one variable)
  • Descriptive analyses for continuous variables
  • Descriptive analyses for discreet/categorical variables
• Two-way descriptive analyses (relationship between two variables)
  • Categorical by categorical
  • Categorical by continuous
  • Continuous by continuous

One-way descriptive analyses

• These are basically what they sound like – the focus is on single variables
• Descriptive analyses for continuous variables
  • means, standard deviations, minima, maxima, counts
• Descriptive analyses for discreet/categorical variables
  • counts, percentages

One-way descriptive analyses: Continuous

soep_long %>% 
  select(Procedural__SID,year,Big5__E:Big5__O) %>%
  pivot_longer(
    cols = contains("Big5")
    , names_to = "trait"
    , values_to = "value"
    , values_drop_na = T
  ) %>%
  group_by(year, trait) %>%
  summarize_at(
    vars(value)
    , lst(mean, sd, min, max)
    , na.rm = T
  ) %>%
  ungroup()

# A tibble: 15 × 6
    year trait    mean    sd   min   max
   <dbl> <chr>   <dbl> <dbl> <dbl> <dbl>
 1  2005 Big5__A  5.46 0.984     1     7
 2  2005 Big5__C  5.90 0.956     1     7
 3  2005 Big5__E  4.82 1.15      1     7
 4  2005 Big5__N  3.95 1.23      1     7
 5  2005 Big5__O  4.51 1.22      1     7
 6  2009 Big5__A  5.35 0.985     1     7
 7  2009 Big5__C  5.82 0.954     1     7
 8  2009 Big5__E  4.77 1.15      1     7
 9  2009 Big5__N  3.82 1.22      1     7
10  2009 Big5__O  4.40 1.22      1     7
11  2013 Big5__A  5.41 0.955     1     7
12  2013 Big5__C  5.84 0.915     1     7
13  2013 Big5__E  4.87 1.11      1     7
14  2013 Big5__N  3.75 1.21      1     7
15  2013 Big5__O  4.60 1.18      1     7

One-way descriptive analyses: Categorical / Count

soep_long %>%
  select(Procedural__SID, contains("_ever")) %>%
  distinct() %>%
  pivot_longer(
    cols = contains("LifeEvent")
    , names_to = "event"
    , values_to = "value"
    , values_drop_na = T
  ) %>%
  group_by(event, value) %>%
  tally() %>%
  group_by(event) %>%
  mutate(total = sum(n)
         , perc = n/total*100)

# A tibble: 20 × 5
# Groups:   event [10]
   event                     value     n total  perc
   <chr>                     <dbl> <int> <int> <dbl>
 1 LifeEvent__ChldBrth_ever      0 19957 22070 90.4 
 2 LifeEvent__ChldBrth_ever      1  2113 22070  9.57
 3 LifeEvent__ChldMvOut_ever     0 19760 22070 89.5 
 4 LifeEvent__ChldMvOut_ever     1  2310 22070 10.5 
 5 LifeEvent__DadDied_ever       0 20952 22070 94.9 
 6 LifeEvent__DadDied_ever       1  1118 22070  5.07
 7 LifeEvent__Divorce_ever       0 21569 22070 97.7 
 8 LifeEvent__Divorce_ever       1   501 22070  2.27
 9 LifeEvent__Married_ever       0 20331 22070 92.1 
10 LifeEvent__Married_ever       1  1739 22070  7.88
11 LifeEvent__MomDied_ever       0 20956 22070 95.0 
12 LifeEvent__MomDied_ever       1  1114 22070  5.05
13 LifeEvent__MoveIn_ever        0 20455 22070 92.7 
14 LifeEvent__MoveIn_ever        1  1615 22070  7.32
15 LifeEvent__NewPart_ever       0 15089 16471 91.6 
16 LifeEvent__NewPart_ever       1  1382 16471  8.39
17 LifeEvent__PartDied_ever      0 21609 22070 97.9 
18 LifeEvent__PartDied_ever      1   461 22070  2.09
19 LifeEvent__SepPart_ever       0 20572 22070 93.2 
20 LifeEvent__SepPart_ever       1  1498 22070  6.79

Two-way descriptive analyses

• Aims to capture relationships between variables

• Categorical by categorical

  • cross-tabs, percentages

• Categorical by continuous

  • means, standard deviations, etc. within categories

• Continuous by continuous

  • correlations, covariances, etc.

Two-way descriptive analyses: Categorical x Continuous

soep_twoway %>%
  group_by(trait, event, occurred) %>%
  summarize_at(
    vars(value)
    , lst(mean, sd, min, max)
    , na.rm = T
  ) %>%
  ungroup() %>%
  pivot_wider(
    names_from = trait
    , values_from = c(mean, sd, min, max)
  )

# A tibble: 20 × 22
   event            occurred mean_Big5__A mean_Big5__C mean_Big5__E mean_Big5__N
   <chr>            <chr>           <dbl>        <dbl>        <dbl>        <dbl>
 1 LifeEvent__Chld… Event            5.48         5.81         4.93         3.84
 2 LifeEvent__Chld… No Event         5.46         5.91         4.81         3.97
 3 LifeEvent__Chld… Event            5.48         6.06         4.87         3.96
 4 LifeEvent__Chld… No Event         5.45         5.87         4.81         3.95
 5 LifeEvent__DadD… Event            5.39         5.91         4.91         3.97
 6 LifeEvent__DadD… No Event         5.46         5.90         4.81         3.95
 7 LifeEvent__Divo… Event            5.32         5.90         5.01         4.08
 8 LifeEvent__Divo… No Event         5.46         5.90         4.81         3.95
 9 LifeEvent__Marr… Event            5.44         5.80         4.99         3.88
10 LifeEvent__Marr… No Event         5.46         5.91         4.80         3.96
11 LifeEvent__MomD… Event            5.49         6.06         4.80         4.01
12 LifeEvent__MomD… No Event         5.46         5.89         4.82         3.95
13 LifeEvent__Move… Event            5.43         5.73         5.08         3.95
14 LifeEvent__Move… No Event         5.46         5.91         4.80         3.95
15 LifeEvent__NewP… Event            5.32         5.56         5.14         3.90
16 LifeEvent__NewP… No Event         5.48         5.97         4.84         3.94
17 LifeEvent__Part… Event            5.61         6.03         4.72         4.12
18 LifeEvent__Part… No Event         5.45         5.90         4.82         3.95
19 LifeEvent__SepP… Event            5.33         5.76         5.03         4.03
20 LifeEvent__SepP… No Event         5.47         5.91         4.80         3.95
# ℹ 16 more variables: mean_Big5__O <dbl>, sd_Big5__A <dbl>, sd_Big5__C <dbl>,
#   sd_Big5__E <dbl>, sd_Big5__N <dbl>, sd_Big5__O <dbl>, min_Big5__A <dbl>,
#   min_Big5__C <dbl>, min_Big5__E <dbl>, min_Big5__N <dbl>, min_Big5__O <dbl>,
#   max_Big5__A <dbl>, max_Big5__C <dbl>, max_Big5__E <dbl>, max_Big5__N <dbl>,
#   max_Big5__O <dbl>

Attributions

Parts of Part 1 of these slides was adapted from Ozan Jaquette’s EDUC 260A at UCLA.