Introduction
This post is written for an Elm-y audience, but might be of interest to other developers too. We’re diving into defining clear application boundaries, so if you’re a believer in miscellaneous middleware and think DRY principles sometimes lead people astray, you may enjoy reading.
Obviously-correct decoders can play a primary role in supporting a changing backend API. Writing very simple decoders pushes transformations on incoming data into a separate function, creating a boundary between backend and frontend representations of the data. This boundary makes it possible to modify server data and Elm application modeling independently.
Decoders
In Elm, Decoders & Encoders provide the way to translate into and from Elm values. Elm is type safe, and it achieves this safety in a dynamic world by strictly defining one-to-one JSON translations.
An example inspired by NoRedInk’s Writing platform follows. We ask students to highlight the claim, evidence, and reasoning of a paragraph in exercises, in their peers’ work, and in their own writing; we need to be able to encode, persist, and decode the highlighting work that students submit.
import Json.Decode exposing (..)
import Json.Decode.Pipeline exposing (..) -- This is the package NoRedInk/elm-decode-pipeline
{-| HighlightedText describes the "shape" of the data we're producing.
`HighlightedText` is also a constructor. We can make a HighlightedText-type record by
giving HighlightedText a Maybe String followed by a String--this is actually how decoders work
and the reason that decoding is order-dependent.
-}
type alias HighlightedText =
{ highlighted : Maybe String
, text : String
}
{-| This decoder can be used to translate from JSON,
like {"highlighted": "Claim", "text": "Some highlighted content.."},
into Elm values:
{ highlighted = Just "Claim",
, text = "Some highlighted content..."
}
-}
decodeHighlightedText : Decoder HighlightedText
decodeHighlightedText =
decode HighlightedText
|> required "highlighted" (nullable string)
|> required "text" string
How do we create our model?
We’ve now decoded our incoming data but we haven’t decided yet how it’s going to live in our model. How do we turn this data into a model?
If we directly use the JSON representation of our data in our model then we’re losing out on the opportunity to think about the best design of our model. Carefully designing your model has some clear advantages: you can make impossible states impossible, prevent bugs, and reduce your test burden.
Suppose, for instance, that we want to leverage the type system as we display what is/isn’t highlighted. Specifically, there are three possible kinds of highlighting: we might highlight the “Claim”, the “Evidence”, or the “Reasoning” of a particular piece of writing. Here’s our desired modeling:
type alias Model =
{ writing : List Chunk
}
type Chunk
= Claim String
| Evidence String
| Reasoning String
| Plain String
So now that we’ve carefully designed our Model, why don’t we decode straight into it? Let’s try to write a single combined decoder/initializer for this and see what happens.
import Model exposing (Chunk(..), Model)
import Json.Decode exposing (..)
import Json.Decode.Pipeline exposing (..)
decoder : Decoder Model
decoder =
decode Model
|> required "highlightedWriting" (list decodeChunk)
decodeChunk : Decoder Chunk
decodeChunk =
let
asResult : Maybe String -> String -> Decoder Chunk
asResult highlighted value =
toChunkConstructor highlighted value
in
decode asResult
|> required "highlighted" (nullable string)
|> required "text" string
|> resolve
toChunkConstructor : Maybe String -> String -> Decoder Chunk
toChunkConstructor maybeString text =
case maybeString of
Just "Claim" ->
succeed
succeed
succeed
succeed
fail
The decodeChunk logic isn’t terrible right now, but the possibility for future hard-to-maintain complexity is certainly there. The model we’re working with has a single field, and the highlighted data itself is simple. What happens if we have another data set that we want to use in conjunction with the highlighted text? Maybe we have a list of students with ids and the highlights may have been done by different students, and we want to combine the highlights with the students… It’s not impossible, but it’s not as straightforward as we might want.
So let’s try a different strategy and do as little work as possible in our decoders. Instead of decoding straight into our Model we’ll decode into a type that resembles the original JSON as closely as possible, a type which at NoRedInk we usually call Flags.
import Json.Decode exposing (..)
import Json.Decode.Pipeline exposing (..)
type alias Flags =
{ highlightedWriting : List HighlightedText
}
decoder : Decoder Flags
decoder =
decode Flags
|> required "highlightedWriting" (list decodeHighlightedText)
type alias HighlightedText =
{ highlighted : Maybe String
, text : String
}
decodeHighlightedText : Decoder HighlightedText
decodeHighlightedText =
decode HighlightedText
|> required "highlighted" (nullable string)
|> required "text" string
Note that HighlightedText should only be used as a “Flags” concept. There might be other places in the code that need a similar type but we’ll create a separate alias in those places. This enforces the boundary between the Flags module and the rest of the application: sometimes it’s tempting to “DRY” up code by keeping type aliases in common across files, but this becomes confusing because it ties together modules that have nothing to do with one another if the data that we’re describing differs in purpose. Internal Flags types ought to describe the shape of the JSON. Type aliases used in the Model ought to be the best representation available for application state. Conflating the types that represent these two distinct ideas may eliminate code, but also eliminates some clarity.
We’re not home yet. We now have a Flags type but we’d really like a Model. Let’s write an initializer to bridge that divide.
import Json.Decode exposing (..)
import Json.Decode.Pipeline exposing (..)
{- FLAGS -}
type alias Flags =
{ highlightedWriting : List HighlightedText
}
decoder : Decoder Flags
decoder =
decode Flags
|> required "highlightedWriting" (list decodeHighlightedText)
type alias HighlightedText =
{ highlighted : Maybe String
, text : String
}
decodeHighlightedText : Decoder HighlightedText
decodeHighlightedText =
decode HighlightedText
|> required "highlighted" (nullable string)
|> required "text" string
{- MODEL -}
type alias Model =
{ writing : List Chunk
}
type Chunk
= Claim String
| Evidence String
| Reasoning String
| Plain String
{- CREATING A MODEL -}
init : Flags -> Model
init flags =
{ writing = List.map initChunk flags.highlightedWriting
}
initChunk : HighlightedText -> Chunk
initChunk { highlighted, text } =
text
|> case highlighted of
Just "Claim" ->
Claim
Just "Evidence" ->
Evidence
Just "Reasoning" ->
Reasoning
Just otherString ->
-- For now, let's default to Plain
Plain
Nothing ->
Plain
We’re still doing the same transformation as before but it’s easier to trace data through the initialization path now: We decode JSON to Flags using a very simple decoder and then Flags to Model using an init function with a type that actually shows what transformation is happening. Plus, as we’ll see in the next section, we have more control and flexibility in how we handle the boundary of our Elm application!
Leveraging Decoders
The example code we’ve been using involves modeling a paragraph with three different kinds of highlights. This example is actually motivated by a piece of NoRedInk’s Writing product, in which students highlight the component parts of their own writing. Earlier this year, students were only ever asked to highlight the Claim, Evidence, and Reasoning of paragraph-length submissions. This quarter, we’ve worked to expand that functionality in order to support exercises on writing and recognizing good transitions; on embedding evidence; on identifying speaker, listener, and plot context; and more. But uh-oh–our Writing system assumed that we’d only ever be highlighting the Claim, Evidence, and Reasoning of a paragraph! We’d been storing JSON blobs with strings like “claim” in them as our writing samples!
So what did this mean for us?
- We needed to store our JSON blobs in a new format–the existing format was too tightly-tied to Claim, Evidence, and Reasoning
- We needed to migrate our existing JSON blobs to the new format
- We needed to support reading both formats at the same time
In a world where the frontend application has a strict edge between JSON values and Elm values and a strict edge between Elm values and the Model, this is straightforward.
import Json.Decode exposing (..)
import Json.Decode.Pipeline exposing (..)
type alias Flags =
{ highlightedWriting : List HighlightedText
}
{-| This decoder supports the old and the new formats.
-}
decoder : Decoder Flags
decoder =
decode Flags
|> custom (oneOf [ paragraphContent, deprecatedParagraphContent ])
type alias HighlightedText =
{ highlighted : Maybe String
, text : String
}
paragraphContent : Decoder (List HighlightedText)
paragraphContent =
{- We've skipped including the actual decoder in order to emphasize
that we are easily supporting two radically different JSON blob
formats--it doesn't actually matter what the internals of those blobs are!
-}
field "newVersionOfHighlightedWriting" (succeed [])
deprecatedParagraphContent : Decoder (List HighlightedText)
deprecatedParagraphContent =
field "highlightedWriting" (list deprecatedDecodeHighlightedText)
deprecatedDecodeHighlightedText : Decoder HighlightedText
deprecatedDecodeHighlightedText =
decode HighlightedText
|> required "highlighted" (nullable string)
|> required "text" string
Conclusion
As we’ve seen, it’s easier to reason about data when each transformation of the data is done independently, and using decoders well can help us handle the intermediate modeling moments that are common in software development.
We hope that you’re interested in how NoRedInk’s Writing platform works: We’ve loved working on it and we hope you’ll ask us about it! We’ve gotten to work with some really cool tools and to try out cool architectural patterns (hiii event log strategy with Elm), all while building a pedagogically sound product of which we’re proud. In the meantime, may your modules have clean APIs, your editor run elm-format on save, and your internet be fast.
Tessa Kelly
@t_kelly9
Engineer at NoRedInk
Jasper Woudenberg
@jasperwoudnberg
Engineer at NoRedInk