Use the Database

Kevin Burke

@derivativeburke

burke.services

Who is this talk for?

Node.js monolith serves:

Customer app
Driver app
Warehouse app
Internal site

Problem 1: items being reacquired

Items being reacquired

Driver acquires item
Item transferred to warehouse
Item marked as shipped
Driver reacquires item

Lots of confusion

Ghost items re-appear in driver inventory
User apps show funky UI
Not clear how to resolve

iOS caching problem

Server side

var item = Items.findOne(id)
item.state = 'ACQUIRED'
item.save()

Use a state machine

Naive state machine

var STATES = {
  ACQUIRED: ['SUBMITTED'],
  CANCELED: ['DRAFT', 'SUBMITTED'],
  // ...
}

Naive implementation

var item = Items.findOne(id)
if 'ACQUIRED' not in STATES[item.state]
  throw(new Error("Invalid state transition"))
item.state = 'ACQUIRED'
item.save()

Figuring out the valid states can be really hard

Really hard

You need to get every call site
You might break every call site!
Clients might send a {state: 'X'}
You don't know all of the transitions

Naive implementation v2

var item = Items.findOne(id)
if 'ACQUIRED' not in STATES[item.state]
  log.warn("Potentially invalid transition...")
item.state = 'ACQUIRED'
item.save()

Problem 2: duplicate submissions

Duplicate submissions

T1: Submit pickup A
T2: Submit pickup A
T1: Items.findOne(A)
T2: Items.findOne(A)
T1: Check STATES['SUBMITTED']
T2: Check STATES['SUBMITTED']

Duplicate submissions (cont'd)

T1: item.state = 'SUBMITTED'
T2: item.state = 'SUBMITTED'
T1: item.save()
T2: item.save()
T1: *Attempt to find a driver*
T2: *Attempt to find a driver*

Two drivers, same pickup

Better state machine

UPDATE items SET state='SUBMITTED'
WHERE id = $1
AND state IN ($2, $3)

You can run that 1000 times
and only one will succeed^*

^*Your database may not support CAS

CAS: a fancy way of
saying UPDATE with WHERE

"jepsen compare and set [your database]"

Lesson: if you type "state machine" into NPM you are doing it wrong

Lesson: read-check-modify-save is incorrect/vulnerable

Just try the write

user = User.find({
  email: 'a@example.com'})
if (user) { throw Error(...) }
User.create(...)

Just try the write (cont'd)

try {
  user = User.create({
    email: 'a@example.com'})
} catch (e) {
  // Update, return...
}

Problem 3: Partial Assignment

Partial assignment

try {
  Driver.transition(driverId, 'ASSIGNED');
  Pickup.transition(pickupId, 'ASSIGNED');
} catch (e) { ?? }

Partial assignment (cont'd)

"Driver X is in ASSIGNED but doesn't have any pickups."

Our #1 cause of
oncall incidents

Use transactions^*

Use transactions

BEGIN
Driver.transition(driverId, 'ASSIGNED');
Pickup.transition(pickupId, 'ASSIGNED');
ROLLBACK/COMMIT

^Your ORM may not support transactions
^Your database may not support transactions

github.com/Shyp/pg-transactions

Transactions can go wrong

Can deadlock
(especially if connection pool is full!)
Can degrade MVCC performance
Missing a code path hurts big time
Conn failure on rollback/commit

Conclusion

Conclusions

Use state machines from the start
(Use a WHERE clause with UPDATE)
No read-modify-save
Pick an ORM with transactions

Consistency is hard

burke.services

Thanks!

Kevin Burke

kev.inburke.com

kev@inburke.com

@derivativeburke

These slides are available at:

Use the Database

Kevin Burke

@derivativeburke

burke.services

Who is this talk for?

Node.js monolith serves:

Problem 1: items being reacquired

Items being reacquired

Lots of confusion

iOS caching problem

Server side

Use a state machine

Naive state machine

Naive implementation

Figuring out the valid states can be really hard

Really hard

Naive implementation v2

Problem 2: duplicate submissions

Duplicate submissions

Duplicate submissions (cont'd)

Two drivers, same pickup

Better state machine

You can run that 1000 timesand only one will succeed*

*Your database may not support CAS

CAS: a fancy way ofsaying UPDATE with WHERE

"jepsen compare and set [your database]"

Lesson: if you type "state machine" into NPM you are doing it wrong

Lesson: read-check-modify-save is incorrect/vulnerable

Just try the write

Just try the write (cont'd)

Problem 3: Partial Assignment

Partial assignment

Partial assignment (cont'd)

Our #1 cause ofoncall incidents

Use transactions*

Use transactions

*Your ORM may not support transactions *Your database may not support transactions

github.com/Shyp/pg-transactions

Transactions can go wrong

Conclusion

Conclusions

Consistency is hard

burke.services

Thanks!

Kevin Burke

kev.inburke.com

kev@inburke.com

@derivativeburke

kev.inburke.com/slides/use-the-database

You can run that 1000 times
and only one will succeed^*

^*Your database may not support CAS

CAS: a fancy way of
saying UPDATE with WHERE

Our #1 cause of
oncall incidents

Use transactions^*

^Your ORM may not support transactions
^Your database may not support transactions