Quantcast
Channel: NoRedInk
Viewing all 193 articles
Browse latest View live

One Source of Truth to Rule them All

$
0
0

From one, there were many.

When I started at NoRedInk in March 2013, one of my first goals was to put together a backlog of what we needed to get done before the start of the 2013-2014 school year. We were going to have our first paying customers, and we needed to thrill them. There was already a Pivotal Tracker account and a huge backlog and icebox of issues to sort through, so I started there. I’d used Pivotal Tracker before for personal projects and was happy enough with it. At my last company, we started our Agile experience using Grasshopper and actually switched to using Post-It notes and magnets on a whiteboard.

As we hired up the engineering team, we quickly realized it was difficult to have a good discussion using Pivotal Tracker. Discussions just worked better in GitHub. We looked at a few alternatives that might bridge the gap, but none worked as well as GitHub for our needs, so we did something foolish: we started using both.

At the same time, various parts of the company were using Asana to track lists of this and that. And, there were a bunch of important requirements being discussed via email. As you and I and everyone we know could tell you, keeping all these systems in sync was a real pain.

There needs to be one source of truth. This is just as good advice for product development as it is for software.

From many, there was one.

Obviously, we did not decide to build our own issue tracker. We’re an education company, and like most companies we’re lucky if we get to do one thing well. That wasn’t going to be issue tracking.

I’m lucky to work with brilliant, talented people with a depth of experiences. Half of us have taught in a classroom. Half of us have founded a startup. And when we added a new member to the team in Oct, 2013, he soon took the reins of the situation and gave us the kick in the pants we needed to make everything work in one place. For us, that one place was GitHub.

GitHub was not designed to run Agile sprints, but most of the elements are there. There are no “sprints” but there are “milestones”. Each user story, task, and epic is an “issue”, and we track and link them together using checkboxes and, well, hyperlinks. We use Zenhub to define our backlog and track issues from started to resolved. And while there are no story points, and we have no idea what our velocity is, the truth is we haven’t missed it.

Equally as important, all of our discussions, even non-engineering ones, are now out of email and categorically in GitHub. This has been a tremendous win! We not only have a solid record of the discussions, but in a public forum where anyone with a good idea can contribute.

There’s no silver bullet

Are we saying you should stop what you’re doing and switch over to doing everything in GitHub? Absolutely not! We’re not even saying you should try to have all parts of your company using the same system. We happily use RelateIQ to track customers through our sales pipeline, and we won’t be replacing it with GitHub anytime soon.

But whenever possible, it is valuable to have all steps of a process - in this case product development - in one place. From ideation, to specification, to prioritization, to task breakdown, to implementation, pull request, quality assurance, and all the way through to resolution, we can not only see what we’re building, but see why we decided that was the right thing to build.

Discuss this post on Hacker News


Josh Leven
Director of Product at NoRedInk
github.com/jleven


Switching from immutable.js to seamless-immutable

$
0
0

We like React and Flux at NoRedInk, and awhile back we decided to add immutability to the mix.

Others have written about the general benefits of immutability, but the primary motivating factor for us was its debugging benefit: knowing for certain that a value could not have been accidentally changed sometime after it was instantiated. This saves us a ton of time, because we can instantly rule out many potential culprits that otherwise we would have to spend time investigating.

This is especially beneficial when using Flux, because it lets us implement our stores in a way where accidental updates of the store’s data (outside using its normal API) become nearly impossible.

Questing for a Guarantee

A quick-and-dirty way to guarantee full immutability (as opposed to shallow immutability like Object.freeze, which does not recurse) is to serialize a given data structure as a JSON string. However, this forces you to deserialize again every time you needed to read values from your data structure, which is both clunky and costly.

What we wanted were data structures that were guaranteed to be fully immutable, but which took no more effort to use than their mutable counterparts.

Since we’d been so impressed with React and Flux, our first instinct was to reach for Facebook’s own immutable.js. We tried it out, but soon hit three problems.

Problem 1: Like Object.freeze, the collections are only shallowly immutable. In other words, any mutable objects we place in immutable.js collections remain mutable.

var obj = {foo: "original"};
var notFullyImmutable = Immutable.List.of(obj);

notFullyImmutable.get(0) // { foo: 'original' }

obj.foo = "mutated!";

notFullyImmutable.get(0) // { foo: 'mutated!' }

Partially mutable collections miss out on immutability’s biggest debugging benefit: knowing for certain that the collection could not have been accidentally changed after instantiation.

Problem 2: We had to do a lot of converting back and forth to go from immutable.js collections to vanilla JS collections. It surely wasn’t as bad as the quick-and-dirty “serialize to JSON” hack would have been, but it was a recurring pain nevertheless.

Since immutable.js collections have a very different internal representation than vanilla JS collections, using them with functions expecting vanilla JS data structures requires an explicit conversion step (using immutable.js helper methods) even if the function would not attempt to mutate the data in the collection.

var hobbits = Immutable.fromJS([{name: "Frodo"}, {name: "Samwise"}, {name: "Meriadoc"}])

_.pluck(hobbits, "name") // Runtime exception
_.pluck(hobbits.toJS(), "name") // ["Frodo", "Samwise", "Meriadoc"]

It wasn’t that this took a lot of effort to code, but rather that it was a nuisance to remember. We’d see a runtime exception crop up, remember that we’d decided to use an immutable.js data structure there, and double back to add the conversion step.

This came up not only for third-party libraries, but also for our internal code. It meant more friction when invoking our own preexisting helper functions, and encouraged writing new helper functions in terms of immutable.js - making them more trouble to use in other parts of the code base.

Problem 3: The API had unorthodox, changing opinions on functional programming fundamentals

Discovering we could not call map on an immutable.js collection and then call map again on the result was a real shock - like discovering that evaluating "foo".toString() would for some reason return {stringRepresentation: "foo"}. We assumed this was a bug, because in the course of normal programming you expect toString to return a string, 5 to be an integer, and map to be chainable. Anything that doesn’t follow these well-established semantics deserves a different name.

When we discovered that this was a design decision and not a bug, it was time to part ways. We spent enough time hunting down the consequence of that design decision the first time it bit us, and the fact that the design was eventually reversed was not enough to restore confidence in a library that would necessarily pervade our code base. As the saying goes: “Fool me twice, shame on me.”

Fortunately, trying out immutable.js did help us enumerate what we wanted in an immutables library:

  1. Fully immutable collections: once instantiated, it’s guaranteed that nothing about them can change.
  2. As little effort as possible needed to use them with JS libraries.
  3. No surprises; APIs follow established conventions.

We searched for something that met all these needs and came up empty; thus, seamless-immutable was born.

Integrating with Third-Party Libraries

Since our calls to third-party libraries tend to be non-mutating, we haven’t spent any noticeable amount of time converting data structures when dealing with them.

An unanticipated benefit of this was realizing we could use much of Underscore’s library of functions as normal. When we wanted to use _.max or _.find, we passed them a seamless-immutable array and everything just worked.

An anticipated—and enjoyable!—benefit is that existing debugging tools work swimmingly with seamless-immutable collections. If you run console.log(someComplicatedImmutableObject), the output is straightforward and readable, and includes all the interactive folding arrows we’ve come to expect for objects in the console.

Backing React Components

As long as you’re using React 0.12 or later, these collections also work just fine as a replacement for React components’ props and state values. (Prior versions of React used to mutate the props and state objects they were passed.)

Naturally, using an immutable object for props or state means that setState and setProps no longer work, as they attempt to mutate those values, but you can use merge with replaceState to achieve the same effect.

(We actually found ourselves doing this so often that we wrote a convenience function that builds components with setState and setProps overwritten to just use merge and replaceState behind the scenes.)

Unfortunately, React components themselves must be mutable objects. As such, it’s not possible to call map on a seamless-immutable array to generate an array of React components; seamless-immutable will call Object.freeze on everything it returns, and React components are not designed to work when frozen.

There are a few ways to resolve this. One way is to use asMutable, which returns a vanilla JS array representation of a seamless-immutable array. Another is to use another library’s map function, such as _.map from Underscore. In CoffeeScript, you also can use a list comprehension (forin) instead of map to similar effect.

Performance

Three performance-related features that we lost when switching from immutable.js to seamless-immutable are lazy sequences, batching mutations, and Persistent data structures under the hood.

We haven’t missed them. Lazy sequences are a reasonable tool to have for performance optimization, but even for our largest immutable instances, we have yet to encounter a performance problem in practice that they would solve. The same is true of batching mutations.

Persistent data structures are different, as their performance improvements are passive. Although seamless-immutable does not (and cannot, while maintaining its backwards compatibility with vanilla JS collections) use things like VLists under the hood, its cloning functions—such as merge—only bother to make shallow copies, as shallow and deep copies of immutable values are equivalent.

In practice, this simple passive optimization has been sufficient; we have yet to encounter a performance problem that Bagwell-style persistent data structures would have solved.

Takeaways

We wanted data structures that were guaranteed to be fully immutable, but which took no more effort to use than mutable equivalents. seamless-immutable provided that where the alternatives we investigated did not.

Not only did they work with Flux data stores, they also provided a fine replacement for React components’ state and props objects. Integrating them with existing third-party libraries, even Swiss Army Knives like Underscore, was a breeze.

Performance has been great, and we’ve been using it in production for months without issue.

In short: Worked as expected; installation was hassle-free; would buy again!

Discuss this post on Hacker News


Richard Feldman
Engineer at NoRedInk
github.com/rtfeldman

Don’t Replace CoffeeScript with ES6 Transpilers

$
0
0

The recent surge in popularity of ES6 transpilers such as Babel (formerly called 6to5), combined with the fact that ES6 has adopted several of CoffeeScript’s improvements to JavaScript, has resulted in the meme that CoffeeScript is now obsolete and is due to be replaced. Does that really follow?

Let’s step back a second. At a high level, Babel and CoffeeScript are two programming languages that compile to JavaScript. Importantly, they both compile to a version of JavaScript that you can use today with older versions of Internet Explorer.

They are not alone. There are many, many other languages that do this.

CoffeeScript is notable for being the most popular language that compiles to JavaScript. Babel is notable for being a language whose design is intentionally identical to a version of JavaScript that is not usable today, but which is assumed to be usable without a compilation step at some point in the future.

The blurb on babeljs.io lays it out:

> Babel will turn your ES6+ code into ES5 friendly code, so you can start using it right now without waiting for browser support.

Clearly Babel is marketed as something meaningfully different from CoffeeScript. The logic goes something like this:

  1. We need to support older versions of Internet Explorer, but that means we can’t use all the nice features modern browsers have added to JavaScript.
  2. To solve this, we will add a Babel compilation step to our build process. Thanks to this minor effort, we get to use a nicer version of JavaScript right now!
  3. Someday browsers will catch up, we will no longer need Babel, and we can breathe a collective sigh of relief as we remove the compilation step from our build process.

In other words, Babel is a significant convenience that we should add to our build process as soon as possible, while simultaneously being a significant inconvenience that we should remove from our build process as soon as possible.

When confronted with this logic, I personally feel the need to pick one conclusion or the other: either adding a Babel compilation step is a significant inconvenience or it isn’t. And for my money, it simply isn’t.

In a hypothetical future where I could use ES6 features without a compilation step, I’d still want Babel for the new ES7 hotness that only half the major browsers had natively implemented. Our collective thirst for sharper programming tools will not end with the mass adoption of ES6, I promise you.

So if the Babel compilation step is never going away, and wasn’t much trouble to begin with, then what makes Babel meaningfully different from CoffeeScript? Aren’t they both just two different programming languages that compile to JavaScript?

Yes. Yes they are.

As with any two programming languages, there are pros and cons to each, but they can be compared on the same level as any other two languages.

Some of the tooling for Babel is nicer. For example, you can use many of its ES6 features in a modern browser’s console without needing an extension like CoffeeConsole. But there’s far more to language tooling than what browsers natively implement; consider Elm’s time-traveling debugger, which we can hardly expect will ever appear on the JavaScript roadmaps of any major browsers.

Some of the features in CoffeeScript are nicer. Everything is an expression, a benefit which would so severely break backwards compatibility that there’s little chance any future version of JavaScript will introduce it. Of course, some of the features in Babel are nicer too; let and const are both useful, and CoffeeScript has no plans to incorporate them.

Again, tooling and features are normal ways to compare two programming languages. There’s nothing intrinsically wiser about choosing Babel over CoffeeScript, because Babel isn’t special. It’s just another language that compiles to JavaScript, with its own unique set of pros and cons.

And that’s exactly how we should treat all ES6 transpilers. Decide whether to adopt them based on their merits as languages that compile to JavaScript, and nothing else.

Discuss this post on Hacker News


Richard Feldman
Engineer at NoRedInk
github.com/rtfeldman

Walkthrough: Introducing Elm to a JS Web App

$
0
0

When we introduced Elm to our production front-end at NoRedInk, we knew what to expect: the productivity benefits of an incredibly well-built language, and the intrinsic risk of integrating a cutting-edge technology for the first time. Knowing these risks, we came up with a minimally invasive way to introduce Elm to our stack, which let us get a sense of its benefits without having to make a serious commitment.

Since several people have asked me about this, I’d like to share how we did it!

This walkthrough assumes only a beginner level of Elm familiarity. I won’t be spending much time on syntax (though I will link to the syntax guide as a reference) or functional programming staples like map and filter, so I can go into more depth on introducing Elm to an existing JS codebase.

Let’s dive in!

Elm for…business logic?

Elm is known for its UI-crafting capabilities (both as an improvement over React as well as for game programming), but building out an entire UI in a new language was more commitment than we wanted to tackle as a starting point. We didn’t want a drink from the fire hose; we just wanted a little sip.

Fortunately, it turns out Elm is also a phenomenal language for writing business logic.

Besides making it easy to write clear, expressive code, Elm also makes it easy to write code that’s rock-solid reliable. How reliable? Here are some numbers:

  • Students answer over 2.5 million questions per day on NoRedInk. Needless to say, edge cases happen.
  • 3% of our code base (so far!) is Elm code. Not a ton, but a substantial chunk.
  • We have yet to see our production Elm code cause a single runtime exception.

This is not because we sheltered it; our production Elm code implements some of the most complicated parts of our most mission-critical features. Despite that level of exposure, every runtime browser exception we’ve ever encountered—in the history of the company—has been fixed by changing CoffeeScript or Ruby code. Never by changing Elm code.

This is also not because we have some particularly unusual code base; rather, it’s because Elm’s compiler is astonishingly helpful.

Let’s get a taste for that by introducing Elm to an existing JavaScript code base!

Adding a hint of Elm to a JS TodoMVC app

We’ve been using React and Flux at NoRedInk since they came out in 2014.

Flux stores are a convenient place to introduce Elm to your code base, because they’re all about business logic and are not concerned with UI rendering. Porting a single Flux store to Elm proved a great starting point for us; it was enough to start seeing real benefits, but not such a big commitment that we couldn’t have changed our minds and turned back if we’d wanted to.

Let’s walk through doing the same thing: porting a store from the Flux TodoMVC example to Elm.

First, grab Facebook’s Flux repo. (I’m linking to a fork of their repo, in case theirs has changed since I wrote this.) and cd into its directory. At this point you should be able to follow the instructions for building the TodoMVC example.

git clone git@github.com:rtfeldman/flux.git
cd examples/flux-todomvc
npm install
npm start

Once you have npm start running, you should be able to open the index.html file in your current directory and see a working app.

If you’re not familiar with the Flux architecture (it’s conceptually similar to The Elm Architecture), now might be a good time to read about it. You don’t need a ton of experience with it to follow along, but having a general idea what’s going on should help.

Now you’ll want to grab yourself a copy of Elm. Either use npm install -g elm or visit elm-lang.org/install to get an installer. When you’re finished, run elm make --help in your terminal and make sure it says Elm Platform 0.15.1 at the top!

Integrating Elm at a basic level requires three things:

  1. A .elm file containing our Elm code.
  2. An elm-package.json file specifying our dependencies.
  3. A JavaScript command to invoke our Elm code.

Let’s create examples/flux-todomvc/TodoStore.elm with the following code:

module TodoStore where

port dispatchCreate : Signal String

Now let’s build it by running this command in the examples/flux-todomvc/ directory:

$ elm make TodoStore.elm --output elm-todomvc.js

The first time you build, you should see this prompt:

Some new packages are needed. Here is the upgrade plan.

 Install:

 elm-lang/core 2.1.0

Do you approve of this plan? (y/n)

After you answer y, Elm will install your dependencies. When it’s done, you should see this message:

Successfully generated elm-todomvc.js.

So far, so good!

Your directory should now contain an elm-package.json file and the compiled elm-todomvc.js file generated by elm make, as well as an an elm-stuff/ folder which holds downloaded packages and the like. I always add elm-stuff/ to .gitignore.

Next we need to add our compiled Elm code to the page.

If we’d incorporated gulp-elm into our project’s Gulpfile (see also grunt-elm, elm-webpack-loader, and this Gist for Sprockets integration), then our compiled Elm code would already be included as part of bundle.js. However, we’re avoiding build tool plugins for this example, so we need to tell our index.html file about elm-todomvc.js by adding a new <script> tag above the one that imports "js/bundle.js" like so:

...

  <script src="elm-todomvc.js"></script>
  <script src="js/bundle.js"></script>
</body>

...

All done! Now our Elm code is compiled and available on the page; we’re ready to start porting.

Porting Over The Wiring

(Note: If you’re more interested in business logic than wiring, feel free to skip this part and grab the completed wiring from this Gist so you can move on to the next section. If you’re curious about the details of where this code came from, you can always return here later!)

We’ve written some Elm code, compiled it, and added the compiled code to our page. Now let’s start porting TodoStore.js to Elm.

I’ve found it’s easiest to start by porting over the wiring. That gives us a foundation for a basic runnable app, which makes the rest of the process go more smoothly.

We’ll start with TodoConstants.js, which represents the different types of Actions our program can use. We don’t need a whole separate file for this, nor do we need the keyMirror library; Elm’s built-in union types are perfect for representing Actions, as they let us incorporate payload information along with the Action’s name.

That means we go from this code in TodoConstants.js…

module.exports = keyMirror({
  TODO_CREATE: null,
  TODO_COMPLETE: null,
  TODO_DESTROY: null,
  TODO_DESTROY_COMPLETED: null,
  TODO_TOGGLE_COMPLETE_ALL: null,
  TODO_UNDO_COMPLETE: null,
  TODO_UPDATE_TEXT: null
});

…to this code in TodoStore.elm, right after the  module TodoStore where part:

type Action
    = Create String
    | Complete TodoId
    | Destroy TodoId
    | DestroyCompleted
    | ToggleCompleteAll
    | UndoComplete TodoId
    | UpdateText TodoId String

Note that each of these specifies not only the name of the Action type, but also what kind of payload the action requires. Create requires a String for the text of the item to be created. Complete takes the ID of the TodoItem to mark as completed. We can have as many or as few payload parameters as we like; DestroyCompleted has no payload, and UpdateText takes two values.

Whereas String is a built-in type, TodoId is a custom one I just made up. Let’s define it right below our type Action declaration:

type alias TodoId = Int

This just says “TodoId is equivalent to Int, so anywhere you see a TodoId, treat it as an Int.” This isn’t necessary, but it lets us write more self-documenting code; it’s certainly easier to tell what kind of payload Destroy TodoId accepts than if we’d written Destroy Int instead! This is also handy because if it means later we want TodoId to be a String, we can change this one line instead of having to change it in several places in our Action declaration.

Next we need to define our API for Elm-to-JS communications. We’ll want to define an inbound port for each of the store’s methods, like so:

port dispatchCreate : Signal String
port dispatchComplete : Signal TodoId
port dispatchDestroy : Signal TodoId
port dispatchDestroyCompleted : Signal ()
port dispatchToggleCompleteAll : Signal ()
port dispatchUndoComplete : Signal TodoId
port dispatchUpdateText : Signal (TodoId, String)

Note that these line up very closely with the Action definition we just gave. That’s not a coincidence! We did that because we want to translate these raw inbound JavaScript values into Action instances, so we can work with them more easily later. Let’s do that:

actions =
    Signal.mergeMany
        [
            Signal.map Create dispatchCreate,
            Signal.map Complete dispatchComplete,
            Signal.map Destroy dispatchDestroy,
            Signal.map (always DestroyCompleted) dispatchDestroyCompleted,
            Signal.map (always ToggleCompleteAll) dispatchToggleCompleteAll,
            Signal.map UndoComplete dispatchUndoComplete,
            Signal.map (\(id, text) -> UpdateText id text) dispatchUpdateText
        ]

Here we’re using Signal.mergeMany and Signal.map to go from seven inbound port Signals, each containing raw JavaScript values, to one Signal of Actions, which will be much easier to work with!

Translating JS requests to Elm Actions

Most of these conversions look the same; for example, Signal.map Create dispatchCreate takes dispatchCreate (an inbound port which is represented as a Signal String) and passes its String payload to Create, which gives us back an Action with the appropriate name and payload.

Since DestroyCompleted doesn’t have a payload and therefore doesn’t take an argument, we use always to throw away the argument Signal.map will try to pass it from dispatchDestroyCompleted. Similarly, since UpdateText takes two arguments but dispatchUpdateText produces a single tuple, we use an anonymous function to destructure that tuple and pass its components as separate arguments to UpdateText.

We’re almost there! Now all we need is an initial model and a way to send updates to that model out to TodoStore.js. To do that, let’s add this code right above our current port declarations:

update action model =
    model

initialModel =
    {
        todos = [],
        uid = 1
    }

modelChanges =
    Signal.foldp update initialModel actions

port todoListChanges : Signal (List TodoItem)
port todoListChanges = Signal.dropRepeats (Signal.map .todos modelChanges)

This connects everything together.

Now we have an initial model with an empty list of TodoItem instances (we’ll come back to uid in the next section), we have a modelChanges Signal that represents changes to that model over time, and we have an update function which translates one model to another using an Action it received from all those JS ports we merged together and mapped over.

Finally, we add a single outbound port, todosChanged, which sends a revised todos list out to TodoStore.js every time that part of the model changes.

Onward to the business logic!

Porting the “Create Item” Feature

Now that we have a basic TodoStore.elm with a bunch of wiring ported over, we can start porting logic from TodoStore.js over to TodoStore.elm.

Our goals will be:

  1. Replace all the code in each branch of the AppDispatcher.register handler’s switch statement with one-liner calls to invoke the appropriate Elm code.
  2. Confine our changes entirely to TodoStore.js, which means: a. Don’t alter any other files besides TodoStore.js (except for adding the new TodoStore.elm) b. Don’t change the public-facing API of TodoStore.js, including event-emitting logic. c. End up with the rest of the app continuing to work exactly as it used to

This was what we did when we first introduced Elm at NoRedInk. Having the peace of mind that only one JS file was changing, and that its public-facing API would remain exactly the same, meant that introducing Elm was a very low-risk undertaking.

To do this, let’s start by invoking our Elm code. Add the following to TodoStore.js, right above the call to AppDispatcher.register:

var defaultValues = {
  dispatchCreate: "",
  dispatchComplete: 0,
  dispatchDestroy: 0,
  dispatchDestroyCompleted: [],
  dispatchToggleCompleteAll: [],
  dispatchUndoComplete: 0,
  dispatchUpdateText: [0, ""]
};

var ports = Elm.worker(Elm.TodoStore, defaultValues).ports;

ports.todoListChanges.subscribe(function(updatedTodoList) {
  // Convert from the flat list we're using in Elm
  // to the keyed-by-id object the JS code expects.
  _todos = {};

  updatedTodoList.forEach(function(item) {
    _todos[item.id] = item;
  });

  TodoStore.emitChange();
});

ports.dispatchCreate.send("Gotta port more JS to Elm!");

// Register callback to handle all updates
AppDispatcher.register(function(action) {

If you bring up the app, the page should render with no JavaScript errors. We haven’t wired up any interaction yet, but if the page renders with no errors, we know we’ve connected the ports successfully. If we had (for example) forgotten to specify a default value for one of our ports, we would see an empty Todo list and an error message in the console.

Now let’s finish up Create.

Besides making our code more self-documenting, type alias can also save us from writing boilerplate; the type alias we defined earlier for TodoItem gets us a “constructor function” for free, which is roughly equivalent to the following JavaScript function:

function TodoItem(id, text, complete) {
    return {
        id: id,
        text: text,
        complete: complete
    };
}

So now instead of writing this:

{ id = 42, text = "pick up groceries", complete = False }

…we can write this:

TodoItem 42 "pick up groceries" False

Notice that this automatically-generated function has a lot in common with the handwritten create function in TodoStore.js; so much, in fact, that porting over create wouldn’t really get us much. (It’s not like we needed to keep that funky way of simulating a server-generated id!)

We do, however, need to port over the dispatcher logic. We’ll do that inside our update function from The Elm Architecture, which we will now expand from just returning model to doing the following:

update action model =
    case action of
        Create untrimmedText ->
            let
                text =
                    String.trim untrimmedText
            in
                if String.isEmpty text then
                    model
                else
                    { model |
                        todos <- model.todos ++ [TodoItem model.uid text False],
                        uid <- model.uid + 1
                    }
        _ ->
            model

Since this code uses String.trim, we’ll need to import String below the first line of the file, like so:

module TodoStore where

import String

There are a few syntactic differences between the Elm version of this logic and the JS version - let instead of var, case instead of switch, and record update syntax for the whole { model | todos <- … } part - but the Elm code is accomplishing the same thing as the JS code did.

Note that because we’re using the Action union type we defined earlier, case can not only match on Create, but also extract its payload (recall that we defined Create to have a String payload), and name that payload untrimmedText. Handy!

The main logical change from the Flux version is that (like all Elm functions) update is a stateless function; it neither mutates any shared state nor has any side effects. All it does is accept a model as an argument and return a new one. So rather than mutating the _todos collection when we create a new item, we instead return a new model that has the new item appended. (Note that Elm uses persistent data structures to store its immutable data, which makes operations like this fast and efficient.)

As for uid, it’s our replacement for the original simulate-an-id code. In a production application we’d get our new id values from a server, but since we don’t have a server here, we need to generate them ourselves. It doesn’t matter what our id values are, so long as they’re unique, so we just store a counter called uid in our model and increment it every time we use its current value to represent a unique id.

Finally, the _ -> model at the end provides a default handler for our case expression. Currently Create untrimmedText -> is the only other case we’ve defined, and this essentially says “if we didn’t match any of the previous cases, then just return the original model.”

Now that we’ve specified what a Create action does, we need to replace the current JS implementation with our Elm one. To make that happen, we replace the handler code for TODO_CREATE with a call to send the relevant data to an Elm port instead:

case TodoConstants.TODO_CREATE:
  ports.dispatchCreate.send(action.text);
  break;

We have now completely delegated the logic for creating todo items to Elm. Try it out! You will be able to add todo items to the list.

Just for fun, let’s verify that we’re using TodoStore.elm instead of TodoStore.js for our logic. Comment out this line in TodoStore.elm, and replace it with the one below:

-- todos <- model.todos ++ [TodoItem model.uid text False],
todos <- model.todos ++ [TodoItem model.uid "Hodor" False],

Run elm-make TodoStore.elm --output elm-todomvc.js again to compile the Elm code, and reload index.html. Now whenever you try to enter a todo item, it will disregard your input and add “Hodor” instead.

Note that at this point, the rest of the app continues to function as normal. We’ve replaced one piece of logic with Elm code, but we haven’t broken anything to do so. This is by design! We set out to introduce Elm incrementally to this code base, and that’s exactly what we’re doing.

Let’s revert the “Hodor” line of code to the way it used to be, and while we’re at it, comment out this line that we just used to confirm things were wired up properly:

// ports.dispatchCreate.send("Gotta port more JS to Elm!");

With this foundation in place, we’re ready to port the rest of the store logic to Elm!

Porting the Remaining Features

With this foundation in place, we’re ready to port the rest of the store logic to Elm!

Let’s do Destroy next. Here’s what we need to add to our case expression, right above the _ -> model clause at the end:

Destroy id ->
    let
        todosWithoutId =
            List.filter (\item -> item.id /= id) model.todos
    in
        { model | todos <- todosWithoutId }

And then we replace the implementation in TodoStore.js with a call to a port:

case TodoConstants.TODO_DESTROY:
  ports.dispatchDestroy.send(action.id);
  break;

Pretty simple, right? If you compile your Elm code, the delete button should work as before.

DestroyCompleted can be a one-liner. Here it is on the Elm side:

DestroyCompleted ->
    { model | todos <- List.filter (\item -> not item.complete) model.todos }

…and then on the JS side:

case TodoConstants.TODO_DESTROY_COMPLETED:
  ports.dispatchDestroyCompleted.send([]);
  break;

Note that ports must always accept a value, but since this port has no meaningful information to send, we declare it as a Signal () on the Elm side and then always send [] on the JS side.

Since Complete and UndoComplete share common logic, we benefit from a helper function. Add this to TodoStore.elm, above the case expression:

withComplete complete id item =
    if item.id == id then
        { item | complete <- complete }
    else
        item

This lets us add case clauses for Complete and UndoComplete as quick one-liners:

Complete id ->
    { model | todos <- List.map (withComplete True id) model.todos }

UndoComplete id ->
    { model | todos <- List.map (withComplete False id) model.todos }

See if you can wire these up in TodoStore.js yourself. (Spoiler: it’s the same procedure we used for Destroy!) Afterwards, run elm-make TodoStore.elm --output elm-todomvc.js again and verify that checking and unchecking todo items still works. You should also see a “Clear completed” button appear after you’ve checked one or more items, and clicking it should delete the checked ones.

At this point, all we have left to port are UpdateText and ToggleCompleteAll. Neither of them is doing anything we haven’t seen before, so I’ll just link to their implementations in the final TodoStore.elm, along with the requisite dispatcher code:

case TodoConstants.TODO_TOGGLE_COMPLETE_ALL:
  ports.dispatchToggleCompleteAll.send([]);
  break;

case TodoConstants.TODO_UPDATE_TEXT:
  ports.dispatchUpdateText.send([action.id, action.text]);
  break;

All done!

Takeaways

Here’s the final diff of what we did.

We kept our changes entirely localized to TodoStore.js, and we did not touch a single other .js file. Even better, the app as a whole continued functioning as we were porting over features, so this was a really low-risk change!

Compared to the original TodoStore.js, the final TodoStore.elm has several benefits:

  • Our wiring got more useful:
  • We removed the need for bespoke event emitting. Now we only call emitChange in one place, and Elm’s Signal.dropRepeats ensures we emit change events exactly when we ought to: every time the todo list actually changes, and at no other times.
  • Everything is immutable, so we no longer have to track down accidental-mutation bugs.
  • There are no side effects, so it’s extremely clear which functions can affect which others.
  • The code is now rock-solid. The compiler will catch so many mistakes for us, it’s now extremely unlikely that we’ll accidentally let a runtime exception slip into production.

This is the same procedure we used to port our first Flux store to Elm at NoRedInk. That code has been running in production for months now, and has yet to cause a runtime exception!

Getting that first low-risk change into production gave us the confidence (and motivation!) to expand our use of Elm on future projects. The next major UI feature we shipped was Elm through and through, with elm-html and The Elm Architecture completely replacing React and Flux. It’s been great! That code is so easy to refactor, it gets more casual polish than any other part of our front-end code base; stay tuned for a future post about it.

By the way, if working on Elm code in production appeals to you…we’re hiring!

Discuss this post on Hacker News


Richard Feldman
@rtfeldman
Engineer at NoRedInk

Thanks to Aaron VonderHaar, Charles Comstock, Mike Clark, Jessica Kerr, and
Texas Toland for reading drafts of this.

Building a Live-Validated Signup Form in Elm

$
0
0

In a previous post, I touched on how we gently introduced Elm to our production stack at NoRedInk. But we didn’t stop there! Since that first course of Elm tasted so good, we went back for seconds, then thirds, and by now elm-html and The Elm Architecture have become our preferred tools for all front-end work.

Where many languages built for the browser offer incremental improvements over JS, Elm’s core design is fundamentally different—like React and Flux taken to a whole new level. This can make parts of the language unfamiliar at first, but it enables things like a time-traveling debugger, a package system where semantic versioning is automatically enforced for every package, and large, complex applications whose Elm code has yet to throw a runtime exception.

Let’s explore these benefits by building a live-validated signup form in Elm!

Rendering a View

If you already have npm, you can get the Elm compiler by running npm install -g elm (or alternatively you can just download the Elm installer). Afterwards you should be able to run this command and see the following output:

$ elm make --version

elm-make 0.16 (Elm Platform 0.16.0)

…followed by some help text. Make sure the Elm Platform version is 0.16.0!

Let’s start by rendering the form. Where JavaScript has Handlebars, CoffeeScript has Jade, and React has JSX, Elm also has its own declarative system for rendering elements on a page. Here’s an example:


view model =
    form [ id "signup-form" ] [
        h1 [] [ text "Sensational Signup Form" ],
        label [ for "username-field" ] [ text "username: " ],
        input [ id "username-field", type' "text", value model.username ] [],
        label [ for "password"] [text "password: " ],
        input [ id "password-field", type' "password", value model.password ] [],
        div [ class "signup-button" ] [ text "Sign Up!" ]
    ]

Like any templating system, we can freely substitute in values from the model. However, unlike Handlebars, Jade, and JSX, this is not a special templating library with its own special syntax - it’s just vanilla Elm code! (You can check out the Elm Syntax Reference for more.)

The above code simply defines a function called view with one argument (model). Much like Ruby and CoffeeScript, everything is an expression in Elm, and the last expression in an Elm function is always used as its return value. In other words, the above code would look essentially like this in JavaScript:


function view(model) {
    return form([ id("signup-form") ], [
        h1([], [ text("Sensational Signup Form") ]),
        label([ for("username-field") ], [ text("username: ") ]),
        input([ id("username-field", type_("text"), value(model.username) ], []),
        label([ for("password") ], [ text("password: ") ]),
        input([ id("password-field", type_("password"), value(model.password) ], []),
        div([ class("signup-button") ], [ text("Sign Up!") ])
    ]);
}

Notice that in Elm you call functions with spaces rather than parentheses, and omit the commas that you’d put between arguments in JS. So in JavaScript you’d write:


Math.pow(3, 7)

…whereas in Elm you’d write:


Math.pow 3 7

When using parentheses to disambiguate, you wrap the entire expression like so:


Math.pow (Math.pow 3 7) 4

On a stylistic note, the Elm Style Guide recommends putting commas at the beginning of the line:


view model =
    form
        [ id "signup-form" ]
        [ h1 [] [ text "Sensational Signup Form" ]
        , label [ for "username-field" ] [ text "username: " ]
        , input [ id "username-field", type' "text", value model.username ] []
        , label [ for "password" ] [ text "password: " ]
        , input [ id "password-field", type' "password", value model.password ] []
        , div [ class "signup-button" ] [ text "Sign Up!" ]
        ]

When I first started using Elm I thought this looked weird, and kept using the trailing-commas style I was accustomed to. Eventually I noticed myself spending time debugging errors that turned out to have been caused because I’d forgotten a trailing comma, and realized that with leading commas it’s blindingly obvious when you forget one. I gave this style a try, got used to it, and now no longer spend time debugging those errors. It’s definitely been worth the adjustment. Give it a shot!

Since Elm compiles to a JavaScript file, the next step is to use a bit of static HTML to kick things off. Here’s a sample page that includes a basic HTML skeleton, some CSS styling, a <script> tag to import our compiled .js file (the Elm standard library is about the size of jQuery), and the one line of JavaScript that kicks off our Elm app:


Elm.fullscreen(Elm.SignupForm);

We’re choosing to have the Elm app use the whole page here, but we could also have it render into a single element—for example, to add a touch of Elm to an existing JavaScript code base.

Finally we need to surround our view function with some essentials that every Elm program needs: naming our module, adding relevant imports, and declaring our main function. Here’s the resulting SignupForm.elm file, with explanatory comments:

SignupForm.elm


module SignupForm where
-- declares that this is the SignupForm module, which is how other modules
-- will reference this one if they want to import it and reuse its code.


-- Elm’s "import" keyword works mostly like "require" in node.js.
-- The “exposing (..)” option says that we want to bring the Html module’s contents
-- into this file’s current namespace, so that instead of writing out
-- Html.form and Html.label we can use "form" and "label" without the "Html."
import Html exposing (..)

-- This works the same way; we also want to import the entire
-- Html.Events module into the current namespace.
import Html.Events exposing (..)

-- With this import we are only bringing a few specific functions into our
-- namespace, specifically "id", "type'", "for", "value", and "class".
import Html.Attributes exposing (id, type', for, value, class)


view model =
    form
        [ id "signup-form" ]
        [ h1 [] [ text "Sensational Signup Form" ]
        , label [ for "username-field" ] [ text "username: " ]
        , input [ id "username-field", type' "text", value model.username ] []
        , label [ for "password" ] [ text "password: " ]
        , input [ id "password-field", type' "password", value model.password ] []
        , div [ class "signup-button" ] [ text "Sign Up!" ]
        ]


-- Take a look at this starting model we’re passing to our view function.
-- Note that in Elm syntax, we use = to separate fields from values
-- instead of : like JavaScript uses for its object literals.
main =
    view { username = "", password = "" }

The starting model we’re passing to the view is:


{ username = "", password = "" }

This is a record. Records in Elm work pretty much like Objects in JavaScript, except that they’re immutable and don’t have prototypes.

Let’s build this, shall we?

$ elm make SignupForm.elm

Some new packages are needed. Here is the upgrade plan.

  Install:
    elm-lang/core 3.0.0

Do you approve of this plan? (y/n)

Ah! Before we can build it, first we need to install dependencies.

elm-lang/core is a basic dependency needed by every Elm program, so we should answer y to install it.

Now we should see this:

Downloading elm-lang/core
Packages configured successfully!
I cannot find find module 'Html'.

Module 'SignupForm' is trying to import it.

Potential problems could be:
  * Misspelled the module name
  * Need to add a source directory or new dependency to elm-package.json

Ah, right. Because we’re using elm-html in addition to core, we’ll need to install that explicitly:

$ elm package install evancz/elm-html
To install evancz/elm-html I would like to add the following
dependency to elm-package.json:

    "evancz/elm-html": "4.0.1 <= v < 5.0.0"

May I add that to elm-package.json for you? (y/n)

The package installer wants to create an elm-package.json file for you. How nice of it! Enter y to let it do this.

Next it will tell you it needs a few dependencies in order to install elm-html:

Some new packages are needed. Here is the upgrade plan.

  Install:
    evancz/elm-html 4.0.2
    evancz/virtual-dom 2.1.0

Do you approve of this plan? (y/n)

Again, enter y to install the dependencies as well as elm-html.  When it’s done, you should see this:

Packages configured successfully!

Now that we have our dependencies installed, we’re ready to build.

Some new packages are needed. Here is the upgrade plan.
$ elm make SignupForm.elm

After this completes, you should see:

Successfully generated index.html.

Sure enough, the current folder will now have an index.html file which contains your Elm code compiled to inline JavaScript. By default, elm make generates this complete .html file so that you can get something up and running with minimal fuss, but we’ll compile it to a separate .js file in a moment.

You should also now have an elm-stuff/ folder, which contains cached build data and your downloaded packages - similar to the node_modules/ folder for your npm packages. If you’re using version control and like to put node_modules in your ignore list, you’ll want elm-stuff/ in your ignore list as well.

If you open up index.html in a browser, you’ll see an unstyled, fairly disheveled-looking signup form. Let’s improve on that by splitting out the JavaScript so we can put it in some nicer surroundings.

elm make SignupForm.elm --output elm.js
Success! Compiled 0 modules.
Successfully generated elm.js

The --output option specifies the file (either .html or .js) that will contain your compiled Elm code.

See how it says Compiled 0 modules? This is because elm make only bothers rebuilding parts of your code base that changed, and since nothing changed since your last build, it knows nothing needed to be rebuilt. All it did was output the results of its previous build into elm.js.

Download the HTML boilerplate snippet as example.html, put it in the same directory as your elm.js file, and open it. Since example.html loads a file called elm.js, you should now see a signup form like so:

signup-1

Congratulations! You’ve now built a simple static Elm view and rendered it in the browser.

Next we’ll add some interaction.

Expanding the Model

Let’s add a touch of interactivity: when the user submits an empty form, display appropriate error messages.

First, we’ll introduce some validation error messages to our model, and render them in the view:

view model =
    form
        [ id "signup-form" ]
        [ h1 [] [ text "Sensational Signup Form" ]
        , label [ for "username-field" ] [ text "username: " ]
        , input [ id "username-field", type' "text", value model.username ] []

        , div [ class "validation-error" ] [ text model.errors.username ]
        , label [ for "password" ] [ text "password: " ]
        , input [ id "password-field", type' "password", value model.password ] []

        , div [ class "validation-error" ] [ text model.errors.password ]
        , div [ class "signup-button" ] [ text "Sign Up!" ]
        ]

If we wrote this same logic in JavaScript, on page load we’d get a runtime exception: “cannot read value "username" of undefined.” Oops! We’re trying to access model.errors.username in our view function, but we forgot to alter the initial model in our main function to include an errors field.

Fortunately, Elm’s compiler is on top of things. It will actually figure out that we’ve made this mistake and give us an error at build time, before this error can reach our users!

$ elm make SignupForm.elm --output elm.js

==================================== ERRORS ====================================

-- TYPE MISMATCH ------------------------------------------------ SignupForm.elm

The argument to function `view` is causing a mismatch.

39│     view { username = "", password = "" }
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Function `view` is expecting the argument to be:

    { b | ..., errors : ... }

But it is:

    { ... }

Detected errors in 1 module.

Let’s take a closer look at what it’s telling us.

Function `view` is expecting the argument to be:

    { b | ..., errors : ... }

But it is:

    { ... }

This tells us that our view function is expecting a record with a field called errors (among other fields - hence the ...), but the record we’ve provided does not have an errors field! Sure enough, we totally forgot to include that. Whoops.

The compiler knows this is broken because we pass that record into our view function as the model argument, and the view function then references model.errors.username and model.errors.password - but given the code we’ve written, there’s no possible way model.errors could be defined at that point.

This is a classic example of the Elm compiler saving us from a runtime exception. It’s really, really good at this - so good that Dreamwriter’s entire Elm codebase still hasn’t thrown a single runtime exception!

Now that we know about the problem, we can fix it by adding an errors field like so:

initialErrors =
    { username = "bad username", password = "bad password" }


main =
    view { username = "", password = "", errors = initialErrors }

Let’s rebuild with elm make SignupForm.elm --output elm.js (which should succeed now) and open the browser. We should see this:

signup-2

Incidentally, in a larger project we’d probably use a build tool plugin like grunt-elm or gulp-elm or elm-webpack-loader rather than directly running elm make every time, but for this example we’ll keep things simple.

Before we move on, let’s make initialErrors start out empty, as the user should not see “bad username” before having had a chance to enter one:


initialErrors =
    { username = "", password = "" }

Next we’ll update the model when the user interacts.

Adding Validation Logic

Now that we’re rendering errors from the model, the next step is to write a validation function that populates those errors. We’ll add it right before our main function:


getErrors model =
    { username =
        if model.username == "" then
            "Please enter a username!"
        else
            ""

    , password =
        if model.password == "" then
            "Please enter a password!"
        else
            ""
    }

For comparison, here’s how this would look in (similarly formatted) JavaScript:


function getErrors(model) {
    return {
        username:
            (model.username === ""
                ? "Please enter a username!"
                : ""),

        password:
            (model.password === ""
                ? "Please enter a password!"
                : "")
    };
}

To recap: we have a starting model, a validation function that can translate that model into error strings, and a view that can report those errors to the user. The next step is to wire it all together. When the user clicks Sign Up, we should use all the pieces we’ve built to display any appropriate errors.

How do we do that? There’s a convenient library called StartApp that makes it easy. Install it like so:

$ elm package install evancz/start-app

We’re also going to want Elm Effects later on, so let’s go ahead and grab that too while we’re at it:

$ elm package install evancz/elm-effects

Next let’s import some modules from these new packages at the top of SignupForm.elm, right below the module declaration, like so:


import StartApp
import Effects

StartApp wraps some boilerplate wiring that most Elm applications (following the The Elm Architecture) will use. Let’s replace our main implementation with code that uses StartApp instead:


app =
    StartApp.start
        { init = ( initialModel, Effects.none )
        , update = update
        , view = view
        , inputs = []
        }

main =
    app.html

StartApp.start wires everything up and returns a record we’ll name app, and from there app.html conveniently provides everything we need to implement main.

Now that that’s out of the way, we need to provide StartApp with an update function to update our model. Let’s add it below our getErrors function:


update action model =
    if action.actionType == "VALIDATE" then
        ( { model | errors = getErrors model }, Effects.none )
    else
        ( model, Effects.none )

This update function’s job is to take an Action (a value that describes what we want done) and a model, and return two things:

  1. A new model, with any relevant updates applied
  2. A description of any effects we want done - such as firing AJAX requests or kicking off animations. We’ll dive into using effects later, but for now we’ll stick to no effects: Effects.none

Have a look at what the else branch returns:


    ( model, Effects.none )

Whenever you see a comma-separated list of elements inside parentheses like this, what you’re seeing is a tuple. A tuple is like a record where you don’t bother naming the fields; ( model, Effects.none ) is essentially a variation on { model = model, effects = Effects.none } where fields are accessed by position rather than by name.

Here the else branch returns a tuple containing the original model unchanged, as well as Effects.none - indicating we do not want any effects like AJAX requests fired off. Compare that to the if branch:


    if action.actionType == "VALIDATE" then
        ( { model | errors = getErrors model }, Effects.none )

This says that if our Action record has an actionType field equal to "VALIDATE" then we should return an altered model and (once again) Effects.none. The first element in this tuple is an example of a record update:


    { model | errors = getErrors model }

You can read this as “Return a version of model where the errors field has been replaced by the result of a call to getErrors model.” (Remember that records are immutable, so an “update” doesn’t actually change them; rather, an “update” refers to returning a separate record which has had the update applied.)

Since our view is already displaying the result of model.errors, and getErrors returns an appropriate list of errors, all that’s needed to perform the validation is to pass this update function an Action record like { actionType = "VALIDATE", payload = "" } and we’ll end up displaying the new errors!

Now we’re ready to connect what we’ve done to a click handler to our submit button. We’ll change it from this:


    , div [ class "signup-button" ] [ text "Sign Up!" ]

…to this:


    , div [ class "signup-button", onClick actionDispatcher { actionType = "VALIDATE", payload = "" } ] [ text "Sign Up!" ]

onClick takes two arguments: an Action - in this case the plain old record { actionType = "VALIDATE", payload = "" } - and somewhere useful to send that action - in this case actionDispatcher. When the user clicks this button, the record { actionType = "VALIDATE", payload = "" } will be sent to actionDispatcher, which is what StartApp uses to connect our view function to our update function. Since StartApp will be passing this dispatcher to our view function, we need to modify view to accept it as an argument:


view actionDispatcher model =

Finally, we need to delegate our main logic to StartApp, which we can do like so:


initialModel = { username = "", password = "", errors = initialErrors }


app =
    StartApp.start
        { init = ( initialModel, Effects.none )
        , update = update
        , view = view
        , inputs = []
        }

main =
    app.html

Let’s build and try to use what we’ve put together so far. Alas and alack! As soon as we submit the form, it clears out everything we’ve typed and presents us with validation errors. What gives?

This is because when we submit the form, it re-renders everything based on the values in the current model…and we never actually updated those! No matter how much you type into either the username field or the password field, those values will not make it into the model of their own free will.

We can fix that using the same technique as before. First we’ll add to our update function:


update action model =
    if action.actionType == "VALIDATE" then
        ( { model | errors = getErrors model }, Effects.none )
    else if action.actionType == "SET_USERNAME" then
        ( { model | username = action.payload }, Effects.none )
    else if action.actionType == "SET_PASSWORD" then
        ( { model | password = action.payload }, Effects.none )
    else
        ( model, Effects.none )

Now if this function receives something like { actionType = "SET_USERNAME", payload = "rtfeldman" }, it will update the model’s username field accordingly.

Let’s create an on handler with the input event (since input events fire not only on keyup, but also whenever a text input otherwise changes - e.g. when you right-click and select Cut), which will dispatch one of these new Actions:


    , input
        [ id "username-field"
        , type' "text"
        , value model.username
        , on "input" targetValue (\str -> Signal.message actionDispatcher { actionType = "SET_USERNAME", payload = str })
        ]
        []

There are two new things here.

The first is targetValue, which is an example of a Decoder. A Decoder’s job is to translate a JSON string or raw JavaScript value (in this case the gargantuan Event object) into something simpler and more useful. We can create our own Decoders if we like, but in this case an off-the-shelf one does what we need; targetValue is a Decoder that performs the equivalent of calling event.target.value in JavaScript - in other words, it gets you the text contents of the input field.

The other thing is (\str -> Signal.message…) - this is just Elm syntax for an anonymous function. The JavaScript equivalent of this would be:


    on("input", getTargetValue, function(str) { return Signal.message(actionDispatcher, { actionType = "SET_USERNAME", payload = str }); });

Finally let’s give password the same treatment:

view actionDispatcher model =
    form
        [ id "signup-form" ]
        [ h1 [] [ text "Sensational Signup Form" ]
        , label [ for "username-field" ] [ text "username: " ]
        , input
            [ id "username-field"
            , type' "text"
            , value model.username
            , on "input" targetValue (\str -> Signal.message actionDispatcher { actionType = "SET_USERNAME", payload = str })
            ]
            []
        , div [ class "validation-error" ] [ text model.errors.username ]
        , label [ for "password" ] [ text "password: " ]

        , input
            [ id "password-field"
            , type' "password"
            , value model.password
            , on "input" targetValue (\str -> Signal.message actionDispatcher { actionType = "SET_PASSWORD", payload = str })
            ]
            []
        , div [ class "validation-error" ] [ text model.errors.password ]
        , div [ class "signup-button", onClick actionDispatcher { actionType = "VALIDATE", payload = "" } ] [ text "Sign Up!" ]
        ]

Now whenever the user types in the username field, an Action is created (with its actionType field being "SET_USERNAME" and its payload field being the text the user entered. Remember that Actions are just data; they don’t intrinsically do anything, but rather describe a particular model change so that other logic can actually implement that change.

Try it out! Everything should now work the way we want.

One great benefit of this architecture is that there is an extremely obvious (and guaranteed!) separation between model and view. You never have model updates sneaking into your view, causing hard-to-find bugs, because views aren’t even capable of updating models. Each step of the process is a straightforward transformation from one sort of value to the next: inputs to actions to model to view.

It may not be what you’re used to, but once you get in the Elm groove, debugging and refactoring get way easier. Based on my experience writing production Elm at NoRedInk, I can confirm that these things stay easier as your code base scales.

Asking a server about username availability

Finally we’re going to check whether the given username is available. Rather than setting up our own server, we’re just going to use GitHub’s API; if GitHub says that username is taken, then that’s our answer!

To do this, we’ll need to use the elm-http library to communicate via AJAX to GitHub’s API. Let’s start by installing it:

elm package install evancz/elm-http

Up to this point we haven’t written any code that deals with effects. All of our functions have been stateless; when they are given the same inputs, they return the same value, and have no side effects.

Now we’re about to…continue that trend! Not only does Elm represent Actions as data, it represents effects as data too—specifically, using values called Tasks. Tasks have some things in common with both Promises and callbacks from JavaScript. Like Promises, they can be easily chained together and have consistent ways of representing success and failure. Like callbacks, instantiating Tasks doesn’t do anything right away; you can instantiate a hundred Tasks if you like, and nothing will happen until you hand them off to something that can actually run them. Like both Promises and callbacks, Tasks can represent either synchronous or asynchronous effects.

The Elm Effects library lets us run Tasks in a way that always results in an Action being fed back into our update function, which means we can neatly incorporate effects into the architecture we’ve been following this entire time. The only difference will be that sometimes instead of returning Effects.none in our tuple, we’ll return Effects.task instead.

Let’s update our "VALIDATE" action to fire off a request to GitHub’s API. In normal Elm code we wouldn’t be this verbose about it, but for learning purposes we’ll break everything down into small pieces.


if action.actionType == "VALIDATE" then
    let
        url =
            "https://api.github.com/users/" ++ model.username

        usernameTakenAction =
            { actionType = "USERNAME_TAKEN", payload = "" }

        usernameAvailableAction =
            { actionType = "USERNAME_AVAILABLE", payload = "" }

        request =
            Http.get (succeed usernameTakenAction) url

        neverFailingRequest =
            Task.onError request (\err -> Task.succeed usernameAvailableAction)
    in
        ({ model | errors = getErrors model }, Effects.task neverFailingRequest)

That’s a decent bit of new stuff, so let’s unpack it!

First, the conditional is now using a let expression. These allow you to make local declarations that are scoped only to the let expression; the rest of our program can’t see usernameAvailableAction or request, but anything between let and the expression just after in can see them just fine. The whole let expression ultimately evaluates to the single expression after in.

Since url, usernameTakenAction, and usernameAvailableAction are all things we’ve seen before, let’s look at Http.get next. It takes two arguments: a Decoder and a URL string. The URL is what you’d expect, and the decoder is like targetValue from our input event handler earlier: it describes a way to translate an arbitrary JavaScript value into something more useful to us.

In this case, we’re not using a very interesting Decoder because any API response error means the username was not found (and thus is available), and any success means the username is taken. Since the Decoder only gets used if the API response came back green, we can use the succeed Decoder, which always decodes to whatever value you give it. (succeed usernameTakenAction means “don’t actually decode anything; just always produce usernameTakenAction no matter what.”)

Finally we have neverFailingRequest. This is important, because the way the Effects library is able to consistently translate a Task into an Action is by requiring that any errors be handled up front. Like JavaScript Promises, Elm Tasks can either succeed or fail, and Http.get returns a Task that can most definitely fail (with a network error, 404 Not Found, and so on). Since Effects.task requires a Task that can never fail, we need to handle that failure case somehow.

Fortunately, the Task.onError function lets us translate any task failure into a success - in this case, a success where we produce usernameAvailableAction to report that the username is available. Since every failure is now translated into a success, we can give Effects.task what it requires: a Task that can never fail. We’re all set!

Now when we enter a username, if that username is available on GitHub (meaning the Http.get results in a 404 Not Found error and thus a failed Task), the result is { actionType = "USERNAME_AVAILABLE", payload = "" } being passed to our update function. If the username is taken (meaning the Http.get resulted in a successful Task), the result is that our “always results in usernameTakenTask” Decoder causes { actionType = "USERNAME_TAKEN", payload = "" } to be passed to our update function instead.

Let’s incorporate these new actions into our update function:


update action model =
    ...

    else if action.actionType == "USERNAME_TAKEN" then
        ( withUsernameTaken True model, Effects.none )
    else if action.actionType == "USERNAME_AVAILABLE" then
        ( withUsernameTaken False model, Effects.none )
    else
        ( model, Effects.none )


withUsernameTaken isTaken model =
    let
        currentErrors =
            model.errors

        newErrors =
            { currentErrors | usernameTaken = isTaken }
    in
        { model | errors = newErrors }

Before we can use all these fancy new libraries, we’ll need to import them at the top, once again below our module declaration:


import Http
import Task exposing (Task)
import Json.Decode exposing (succeed)

Remember how Tasks don’t actually do anything until you hand them off to something that can run them? StartApp takes care of packaging up the Tasks that come out of our update function into something that’s easily runnable (and exposes it as app.tasks), but to actually run them we need to add this to the end of our file:


port tasks : Signal (Task Effects.Never ())

port tasks =
    app.tasks

Ports are a subject for another post, but for now it’s enough to know that this will get us what we want. Finally, we need to introduce the “username taken” validation error to initialErrors, getErrors, and view:


initialErrors =
    { username = "", password = "", usernameTaken = False }


getErrors model =
    …
    , usernameTaken = model.errors.usernameTaken
    }


view actionDispatcher model =
    …
    , div [ class "validation-error" ] [ text (viewUsernameErrors model) ]


viewUsernameErrors model =
   if model.errors.usernameTaken then
       "That username is taken!"
   else
       model.errors.username

With the comments removed, here’s our final result.

Now the username field is validated for availability, and everything else still works as normal. We did it!

Wrapping Up

We’ve just built a live-validated signup form with the following characteristics:

  • Clear, obvious, and strict separation between model and view
  • A compiler that told us about type errors before they could become runtime exceptions
  • High-performance reactive DOM updates like React and Flux
  • Live AJAX validation of whether a username is available
  • Immutability everywhere
  • Not a single function with a side effect

To save time, we didn’t do everything perfectly idiomatically. The fully idiomatic approach would have been to split this up into several files, to represent Actions using the more concise and robust union types and case expressions instead of records and if/then/else expressions, and to document various functions using type annotations.

I plan to do a follow-up post explaining how to get from this code to the idiomatic version, and expanding on this example to show techniques like:

  • Using the contents of AJAX responses in business logic
  • Using fields with different types of inputs - checkboxes, dropdowns, etc.
  • Integrating a third-party JavaScript library (mailcheck) for suggesting misspelled email addresses
  • Submitting the form two ways: either using AJAX or using direct form submission
  • Sending anti-CSRF authenticity tokens along with AJAX requests

Stay tuned!

By the way, if working on Elm code in production appeals to you…we’re hiring!


Richard Feldman
@rtfeldman
Engineer at NoRedInk

Thanks to Joel Clermont, Johnny Everson, Amjith Ramanujam, Jonatan Granqvist, Brian Hicks, Mark Volkmann, and Michael Glass for reading drafts of this.

Optimizing MySQL for high concurrency on Amazon RDS

$
0
0

transaction response timeNoRedInk’s degraded performance before fixing MySQL bottlenecks

TL;DR

MySQL couldn’t handle our load regardless of our server capacity. We fixed it by disabling the query cache, removing the thread concurrency limit and increasing the number of concurrent open tables.

Given that we’re in the education space, NoRedInk has an interesting traffic pattern unlike most other web applications. They are:

  • High traffic that starts around 8:00am ET and dies down at around 5:00pm PT on workdays.
  • Low traffic during nights and weekends.
  • Low traffic during summer break.

This means that, although we have daily “traffic spikes”, our infrastructure is setup to handle them automatically. Given how different our high traffic demands are from low traffic and also that we have an opportunity to tune scaling daily, we feel like we have the tuning aspect of scaling mostly nailed by now.

The fact that our traffic dies down during summer break, though, means that we have about a month of silence where our actual traffic demands increase but are invisible to us. When summer break ends, we’re faced with a bottleneck that we would have eventually found and gradually fixed otherwise.

That’s what happened this summer break to our MySQL database: traffic increased more than we anticipated, MySQL started failing in ways that we didn’t understand, and the metrics pointed us nowhere.

Our memory usage? Under control, no swapping and with spare memory. Disk IO? way under capacity. CPU usage? Aside from a few peaks, under half capacity. Network IO? No visible bottleneck.

MySQL comes with some really poor default configs, but Amazon RDS updates most of those to good defaults. That meant that once we started running into trouble we ruled out looking into those parameters at first. It ends up the configuration we had was the culprit, however, and that updating some of those values allowed us to unleash the full potential of the hardware backing our database.

I’m sure you’re curious to learn more! Let’s get busy.

Looking for concurrency bottlenecks

When you run into problems with your database performance, the first step is usually to check if your machine has any bottlenecks on CPU, memory or IO from either disk or network. Generally, one of those is the culprit, and increase its capacitiy solves the problem. But what do you do when that’s not the case?

For high concurrency scenarios with MySQL, a bottleneck with no clear hardware capacity culprit will usually occur when there’s a mutex causing contention somewhere, or when a particular parameter is blocking the number of concurrent connections that can access a particular part of the internal storage engine. Unfortunately, in cases like this, it will be difficult for you to diagnose the actual problem. If there are multiple bottlenecks, it is hard to debug them in isolation, which makes identifying them even harder.

Your best bet for a first step when trying to understand where your database is spending its time is by calling SHOW PROCESSLIST. This will show what each of your threads are doing at the moment. If a significant portion of them are on the same state, for instance “Waiting for query cache lock” or “Opening tables”, you will then know where to dig deeper.

The best tool you have at your disposal in order to find more detailed information about those bottlenecks is the performance schema. You will get access to a number of different metrics once it’s enabled that would have been unavailable otherwise.

For our debugging, in most cases what we did was one of two things:

1) List the total maximum wait by event inside of InnoDB.

-- "example of tracking down exactly what the contention points are"
SELECT EVENT_NAME,
       SUM_TIMER_WAIT/1000000000 WAIT_MS,
       COUNT_STAR
  FROM performance_schema.events_waits_summary_global_by_event_name
  ORDER BY SUM_TIMER_WAIT DESC, COUNT_STAR DESC LIMIT 30;


-- table handler waits on top, they also had the biggest increase through time
+------------------------------------------------------+-----------------+--------------+
| EVENT_NAME                                           | WAIT_MS         | COUNT_STAR   |
+------------------------------------------------------+-----------------+--------------+
| idle                                                 | 4661443248.1257 |   6165953390 |
| wait/io/table/sql/handler                            | 1755211603.3699 | 381701231771 |
| wait/io/file/innodb/innodb_log_file                  |  459199280.6118 |    139800252 |
| wait/io/file/innodb/innodb_data_file                 |   83050382.6978 |    197884296 |
| wait/io/file/myisam/kfile                            |   56080735.9075 |   5274545307 |
| wait/io/file/myisam/dfile                            |   13172549.9320 |    725383142 |
| wait/lock/table/sql/handler                          |    5669784.9629 |  15013313221 |
| wait/io/file/sql/binlog                              |    2407201.1162 |    389713292 |
...

2) List current and last wait events on InnoDB.

-- "what the current, or last completed, wait for each session was, and for exactly how long they waited"
SELECT NAME,
       IF(PPS.THREAD_ID = 0, 'Internal Thread', CONCAT(IPS.USER, '@', IPS.HOST)) USER,
       DB,
       COMMAND,
       STATE,
       TIME,
       EVENT_NAME LAST_WAIT,
       IF(TIMER_WAIT IS NULL , 'Still Waiting', TIMER_WAIT/1000000000) LAST_WAIT_MS
  FROM performance_schema.events_waits_current
  JOIN performance_schema.threads PPS USING (THREAD_ID)
LEFT JOIN INFORMATION_SCHEMA.PROCESSLIST IPS ON IPS.ID = PPS.THREAD_ID;

-- table handler waits, discrete records:
+-----------------------------------+----------------------------------+----------+---------+--------------+------+-----------------------------------------+---------------+
| NAME                              | USER                             | DB       | COMMAND | STATE        | TIME | LAST_WAIT                               | LAST_WAIT_MS  |
+-----------------------------------+----------------------------------+----------+---------+--------------+------+-----------------------------------------+---------------+
| thread/sql/one_connection         | noredink_app@172.31.35.138:36177 | noredink | Sleep   |              |    1 | idle                                    | Still Waiting |
| thread/sql/one_connection         | noredink_app@172.31.17.171:49280 | noredink | Sleep   |              |  133 | idle                                    | Still Waiting |
| thread/sql/one_connection         | noredink_app@172.31.31.142:54139 | noredink | Sleep   |              |   42 | idle                                    | Still Waiting |
| thread/sql/one_connection         | noredink_app@172.31.4.22:43788   | noredink | Query   | update       |    0 | idle                                    | Still Waiting |
| thread/sql/one_connection         | noredink_app@172.31.33.13:35375  | noredink | Query   | updating     |    0 | wait/io/table/sql/handler               | Still Waiting |
| thread/sql/one_connection         | noredink_app@172.31.42.196:55806 | noredink | Sleep   |              |    4 | idle                                    | Still Waiting |
| thread/sql/one_connection         | noredink_app@172.31.4.22:38855   | noredink | Sleep   |              |   14 | idle                                    | Still Waiting |
| thread/sql/one_connection         | noredink_app@172.31.17.171:55028 | noredink | Sleep   |              |    3 | idle                                    | Still Waiting |
| thread/sql/one_connection         | noredink_app@172.31.19.73:55344  | noredink | Sleep   |              |  229 | idle                                    | Still Waiting |
| thread/sql/one_connection         | noredink_app@172.31.42.196:59682 | noredink | Sleep   |              |   15 | wait/io/table/sql/handler               | 0.1417        |
| thread/sql/one_connection         | noredink_app@172.31.15.149:36555 | noredink | Sleep   |              |   66 | wait/io/table/sql/handler               | Still Waiting |

credit for the queries goes to Mark Leith

For the first one, we’d measure all items across time to see which seemed to peak during the periods where our database performance would have problems. We would then see if they had correlation in increased wait with the periods where we ran into problems.

That’s the backstory. Now here’s what we actually changed so that our MySQL would be perform again.

Disabling the query cache

The query cache is a feature of MySQL that allows it to return the same results when the same query is executed more than once, without having to fetch data and redo its calculations. It writes the results of the first query to the cache, and when a second identical query is executed, it just fetches the results from the cache and returns it directly. This speeds up query execution significantly, especially for queries that are expensive to fetch and compute.

Disabling the query cache is scary. NoRedInk’s usage scenario can mostly be broken down into 2 parts: 1) students answer questions 2) teachers see data about students’ answers. For the reports used to pull data for teachers to see, there are some huge queries. Huge and slow. We were afraid that disabling the query cache could cause more harm then good by removing a bottleneck just to add another.

So you can imagine our surprise when we found out that the query cache was in fact a huge hindrance to our database performance. There were two reasons:

1) If you have a workload that’s write heavy for a particular table, there will be constant invalidation of your cache. Since the cache invalidation happens at the table level, any insert on a particular table will invalidate the entire cache for that table. As quoted from the mysql query cache documentation:

Server workload has a significant effect on query cache efficiency. A query mix consisting almost entirely of a fixed set of SELECT statements is much more likely to benefit from enabling the cache than a mix in which frequent INSERT statements cause continual invalidation of results in the cache.

2) There’s a mutex around the code that handles query cache write, which means only 1 connection writes to the cache at a time. Additionally, when the cache is locked for writing, it applies a read-lock that prevents other connections from reading from the cache. To quote from the mysql query cache configuration documentation

Be careful not to set the size of the cache too large. Due to the need for threads to lock the cache during updates, you may see lock contention issues with a very large cache.

Given those two limitations of the MySQL query cache, it’s likely that, for a big concurrency scenario, the query cache is doing more harm than good. That’s definitely what we found in our case.

If you want to disable the query cache, set the query_cache_size property to 0.

Increasing InnoDB thread concurrency

InnoDB has an internal control for how many threads are allowed to enter it at the same time. InnoDB will keep track of how many threads are inside itself by assigning each a “ticket,” and it also counts the number of tickets that have been handed out. If the number is above the one configured, it’ll add the thread to a queue while it waits for the next available ticket.

There are two primary reasons why you would want to set a limit for thread concurrency:

  • If you have a small installation and are afraid not setting a limit on concurrency will cause it to thrash other machine resources.
  • If you don’t want your database to peg resources allocated for other services running on the same machine.

Given the above, this setting should always be either disabled or set to the maximum value on a high concurrency installation. Surprisingly, it’s set by default to 8, likely because of the number of vCPUs the database originally had.

You might want to set the value to the maximum value instead of disabling the thread concurrency. If the thread concurrency is disabled, InnoDB will no longer collect thread concurrency metrics. To quote from the documentation:

Disabling thread concurrency checking enables InnoDB to create as many threads as it needs. A value of 0 also disables the queries inside InnoDB and queries in queue counters in the ROW OPERATIONS section of SHOW ENGINE INNODB STATUS output.

Setting it to the maximum should cause no visible performance overhead, while still allowing you to access metrics.

Here’s our chart before, with concurrency pegged at 8 concurrent threads (16 CPUs), and after, with the configuration allowing the maximum number of threads. You can see we had 250 threads waiting to execute while the site was under stress with a maximum of 8 threads executing simultaneously. With the limit set to maximum, you can see peaks that go over 50 – meaning that, before, we had hundreds of threads that were blocked from executing.

InnoDB after changeOur InnoDB profile before raising the limit on concurrent threads.

InnoDB before changeOur InnoDB profile after raising the limit.

In order to disable your innodb thread concurrency, set the property innodb_thread_concurrency to 0. In order to keep the metrics for current threads, set it to the maximum, 1000, instead.

Increasing InnoDB open tables

MySQL has an LRU cache for open tables, a cache that stores table metadata for use by each database connection.

When we found out the table cache was a possible bottleneck, we collected some metrics about the number of open tables to check if they were correlated to our performance issues.

Spreadsheet Snippet

One way to see if performance could benefit from open table cache is issuing the SHOW GLOBAL STATUS command, and checking the property Open_tables. If the limit of concurrent open tables is a bottleneck, the value on the property will be the same as the max amount set by table_open_cache. This means some connections are waiting before they can get the necessary information about the table to continue executing.

Finding the ideal max level was more art than science. We slowly increased the limit in increments of one to two thousand. Once the number of currently open tables didn’t reach the maximum we set after about an hour, we figured that we found a good number for our new maximum. Since there are reports online that setting it too high is detrimental to performance, we wanted to make sure it was not much higher than the peak of open tables our database load requires.

In order to set the number of open tables, update the property table_open_cache. We updated ours from 2000 to 8000.

Conclusion

MySQL configuration parameters have a big impact on the overall performance of your application, and tuning it to your workload is incredibly important, especially as you scale. If ignored, you may end up in a position similar to ours where throwing more hardware at the problem had no effect.

Keep in mind this is what worked for us in this particular scenario. Although some parameters might seem like no-brainers, database load and usage profile can vary drastically across applications, and what helps one profile may hurt the other. So be careful!

If you’re excited about infrastructure automation and performance and love debugging interesting issues, we’re hiring operations / infrastructure engineers. You can learn more about our positions on our jobs page.

That’s it! Hope you had a fun read and have a great day!

Discuss this post on Hacker News


Marcos Toledo
@mtoledo
Director of Engineering at NoRedInk

Creating Repeatable CSS Animations with Sass

$
0
0

working animations gif

The problem: an HTML element cycles through one of four possible classes every time it is clicked, each giving a different background color to the element. We want the element to visually pulse when it shifts between states.

Easy enough, right? We’ll just attach a custom CSS animation to each of the four classes, ensuring that an animation fires every time the element changes. Here’s that approach in action (with animations lovingly taken from Animate.css):

So, that didn’t work! Unfortunately, the animation only runs a single time – on first click – and then never again. To understand why it failed, we need to first learn more about modelling a changing element with CSS animations.

CSS Animations

We can do much more complex things using CSS animations (right) but it is much easier to trigger CSS transitions based on state changes (left).

CSS animations allow you to control how an element is rendered to the DOM. At its core, an animation is made up of two parts:

  • A keyframe set, that declares how an element should be displayed at various stages of being rendered. They are constructed as a series of percentages. For instance, the following keyframe describes an animation named “stretch” that expands an element to 200px when it is 30% competed, and then returns the element’s width to 100px when it has finished.

    @keyframes stretch
      0%
        width: 100px
      30%
        width: 200px
      100%
        width: 100px
    
  • A set of CSS styles attached to a CSS selector that determine various details of the animation, such as its timing and duration. Whenever an element matches the specified selector, the associated animation will run. The following code attaches the above stretch animation to any element with the .widget class that is being hovered over. The animation runs for 2 seconds, after 0.3 second delay.

    .widget:hover
      animation-name: stretch
      animation-duration: 2s
      animation-delay: 0.3s
    

The resulting code works like this:

With these building blocks, CSS animations provide us fine-grain control over every step of the rendering process. Combined with the CSS transform property, it is possible to create incredibly intricate animations – check out Animate.css for some amazing animations written purely in CSS.

Reacting to Property Changes

There are downsides, however – CSS animations are built to run immediately after the specified element is rendered, and consequently can be difficult to use when animating a rendered element as it’s CSS properties change. For example, you cannot declare that every time an element’s height changes, a particular animation should fire.

CSS transitions, on the other hand, are much less powerful, but are often better for managing elements that change in predictable ways. To use a CSS transition, you simply declare how a property of an element should transition when changed. For instance:

.widget
  width: 100px
  transition: width 2s linear

.widget:hover
  width: 200px

The above element would take 2s to transition whenever its width value is changed, which in this case would occur when the element is hovered over. The “linear” keyword determines the timing function of the transition. You can learn more about timing functions and how they work on the Mozilla developer docs.

The resulting code would look like this:

In the above example, attaching a transition to the .widget class actually gave us two animations – it automatically animated the element’s change from 100px to 200px, as well as the change from 200px to 100px. If we wanted to do the same with CSS animations, we would have to create two keyframe sets that are functionally inverse of each other, and attach both as unique animations.

When thinking about our original problem, it would relatively easy to model with CSS transitions. We could simply tell the specified HTML element to transition its background-color property:

It works like a charm! Unfortunately, our specification requires us to use a complex animation that cannot be easily constructed using a CSS transition. If we need to create a custom keyframe set, we cannot use the declarative, state-modeling syntax of CSS transitions.

Repeatable Animations

So why did our original code not work? Unlike CSS transitions, it’s difficult to trigger CSS animations based on state changes. Left to its own devices, a CSS animation is not repeatable – it will run once and only once. If an animation is already attached to a given element, the browser will prevent the same element-animation pair from running again. In our case, all four selectors had the same value for the animation-name attribute, so the browser treated them as identical animations.

So then how do we re-trigger an animation that is already attached to the specified element? CSS Tricks has a good rundown of possible strategies. Most answers use JavaScript to trick the browser into thinking an element has changed, using either cloning or setTimeout. There are architectural and performance downsides to those approaches, however, and in general we at NoRedInk try to avoid using JavaScript for animating the DOM.

A CSS-only approach is to simply generate multiple keyframe sets, and attach a different set to each class. While the animations are identical, the browser treats them as distinct, which allows for the animations to be triggered infinitely. While effective, this approach would lead to lots of CSS duplication, as it is impossible in pure CSS to generate multiple keyframe sets in a single declaration.

That’s where Sass comes in. Here’s an implementation of the above strategy using a custom built mixin:

As you can see, we’ve built a unique-pulse helper that creates a unique, but functionally identical, keyframe set and attaches it to each class. It takes advantage of the Sass unique-id function, which we can use to generate a unique animation-name value every time the mixin is called.

Combined with the fact that keyframe set declarations bubble up the tree, regardless of where they are declared, we can generate as many “unique” keyframe sets as necessary. The eventual code looks something like this:

@mixin unique-pulse
  $animation-name: unique-id()
  # pulse animation keyframes
  @keyframes #{$animation-name}
    0%
      transform: scale(1)
    50%
      transform: scale(1.1)
    100%
      transform: scale(1)
  animation-duration: 1s
  animation-timing-function: ease
  animation-name: $animation-name

There are some limitations to this approach – all possible states of the element must have their own class, and the element will not animate unless there is a class change. For more control over when the animation is fired, we would have to use JavaScript. That said, this is a relatively simple technique for adding repeatable animations to your application without adding much complexity to your front-end architecture.

If you’re interested in architecting maintainble front-end codebases, NoRedInk would love to work with you! Check out our jobs page for more information.

Discuss this post on Hacker News


Srinivas Rao
@raorao_
Engineer at NoRedInk

Welcome Evan Czaplicki!

$
0
0

We are thrilled to announce Evan Czaplicki, creator of the Elm programming language, is joining NoRedInk as an Open Source Engineer!

What will Evan’s role be?

Evan will continue to work on the Elm programming language full time. He’ll maintain 100% control over the language, from intellectual property to design decisions.

Additionally, he’ll periodically help the rest of the team at NoRedInk with our most intricate Elm problems and challenges, as well as help us to be even more prolific open source contributors.

You can also expect a few technical posts from Evan himself here on our blog in the future!

NoRedInk is the top Elm company in the world

For those of you who are interested in Elm, it should come as no surprise that NoRedInk is investing in it. We are currently writing all of our new frontend code in Elm and are constantly porting our old code from JavaScript and React into Elm as well.

Our team is also really active in Elm open source and its community. If you participate in any of the Elm channels or mailing lists, it’s really likely that you’ve already met some of us. We also host the San Francisco Elm meetups!

In addition to Evan, NoRedInk has some of the most talented members of the Elm community on its staff, all of whom believe in its potential to make NoRedInk a more disruptive and impactful learning platform.

A note on the importance of supporting open source

In open source communities, there’s been a lot of talk about companies benefiting immensely from open source without giving back.

It’s unprecedented for a company of our size to hire a language creator, and bringing Evan onto our team will give Elm the help it needs to execute on its vision and become the best language and ecosystem it can be.

Want to be a part of this?

We’re on a mission to make learning easy and fun for millions of students. There are many ways for you to use your talent to effect positive change in the world, and education is one of the most exciting spaces for you to do it.

We have an environment with autonomy and freedom. We want to hire the best people and give them the space and time they need to be proud of their work.

If this sounds enticing, come work with us! Our jobs page is always up to date!

Thank you and have an amazing 2016!


Marcos Toledo
@mtoledo
Director of Engineering at NoRedInk


Five Quick Tips for Submitting Conference Talks

$
0
0

NoRedInkers love submitting talk proposals to conferences. It give us the opportunity to share our ideas with larger programming communities. That earns us valuable feedback, plus offers of help and support.

That’s the easy choice. The hard choice usually follows: “What do I say in my proposal?” I have reviewed incoming proposals for four conferences now, so let me share some quick tips based on what I have seen.

There’s No I in A-B-S-T-R-A-C-T

Ban the word I. Also we, me, our, etc. In other words, don’t talk about you.

You’re building a talk for the attendees. Talk to them. Tell them why they should attend your talk and what they will learn from it.

This is a key point that should ripple all the way up to your selection of topics. Are you considering giving a check-out-my-library talk? Vetoed. (Did you catch the problematic word my there?)

Now, that doesn’t mean you can’t give the talk you want to give. You just need to work on your framing. It’s totally valid to show some problems that attendees commonly face and then tell them how your library can be used to solve those problems.

Just remember, the talk is about them, not you.

Strike While the Iron is Luke Warm

If we define iron as the servers hosting the Call For Proposals (CFP) form, then they are plenty cool when the call opens and in danger of catching fire when the buzzer dings. When I was on the program selection committee for RubyConf, we received a little under 400 talk proposals with over 100 of those coming in during the last three days of our two month call.

Try not to be on either end of that range. If you’re too early, your talk could feel like old hat or be forgotten by the time the important decisions are being made. Being at the end is far worse. When a CFP closes, the pressure is on to get the acceptance emails out as soon as possible. All reviewers want to believe we’re as fair as possible, but the reality is that you won’t get the same treatment when you’re the 30th proposal I read that day.

Don’t raise the bar on how much better your proposal will need to be to get accepted. Aim to be in the middle of the call window.

The Goldilocks Zone

Most conferences provide a reviewer’s eyes only field for elaborating on your plans for the talk. This is often called something like Details.

Deciding how much information to include in this field can be tricky. You need to give reviewers a feel for what you want to do, so they can get behind the idea. However, they have to read a lot of proposals and you need to respect their time as well. You don’t want their first impression to be, “Ugh.” Like Goldilocks, you want them to think what you included was, “Just Right.”

Do not summarize your entire talk in this field. It’s too long and you’ll probably change some bits before you give it anyway. Just don’t go there.

An outline is probably the best detail you can include. That gives an overview without going too far. It’s probably also worth explaining, in slightly more detail, the main points or examples. If you feel like you need lengthy background material or chunks of code, provide them as links so a reviewer can choose how much they want to dig.

Bio-ology

I have just a few pieces of advice when it comes to writing a strong speaker bio:

  • Tailor your bio to the specifics of your talk, like you would tailor a good resume to the job
  • Sell the reasons you are qualified to give this talk and skip any perceived weaknesses you feel compelled to mention
  • Add a dash of humor
  • Avoid mentioning identifying information (including GitHub repositories) outside of the designated bio field in case the conference organizers use a blinded review process

Main Title

I’ve asked, and been asked, if a clever title matters. Answers vary, but one thing is certain: it rarely hurts.

If you’re introducing Powerful New Techniques, then a self titled talk is probably fine. Otherwise, I would put some effort into finding a title at least as clever as the subtitles of this blog post.

Good luck with your submissions. We hope to run into you at future conferences.


James Edward Gray II
@JEG2
Engineer at NoRedInk

Static site generation in Elm

$
0
0

Experiments in Elm

This post is part of a mini series based on some of the experiments we’ve been doing at NoRedInk. A lot of these ideas are proof of concepts that aren’t intended to go anywhere. The focus is on experimentation and trying new things out! The majority of these posts will be about Elm.

Static site generation

In this post, we’re going to talk about static site generation. Jekyll? I’d rather use Elm! With elm-static-site, doing this is trivial.

Imagine we have a blog. On our blog, we have multiple bloggers, which we have in a list like this -

-- Blog/Bloggers.elm
module Blog.Bloggers where

type alias Blogger =
    { name : String
    , age : Int
    }

noah =
    { name = "Noah"
    , age = 23
    }

dave =
    { name = "Dave"
    , age = 33
    }

bloggers : List Blogger
bloggers =
    [ noah
    , dave
    ]

Great. Now let’s make a statically generated page to list our bloggers.

-- Blog/Index.elm
module Blog.Index where

import Html exposing (..)
import Blog.Bloggers exposing (..)

bloggerView : Blogger -> Html.Html
bloggerView blogger =
    div
        [ id blogger.name]
        [ text "Hello, my name is "
        , text blogger.name
        , text " and I'm "
        , text (toString blogger.age)
        , " years old!"
        ]

view =
    div
        []
        (List.map bloggerView bloggers)

This will generate a page like this

<div>
    <div id="Noah">
        Hello, my name is Noah and I'm 23 years old!
    </div>
    <div id="Dave">
        Hello, my name is Dave and I'm 33 years old!
    </div>
</div>

For every file in the folder you give it, it will take that file, and if it has a view : Html function, it will take that out and turn into it a static page instead. So let’s say we had this structure:

examples/
    Index.elm -- has a view : Html
    Blog/
        Index.elm -- has a view : Html
        Bloggers.elm -- doesn't have a view : Html

the following files will be generated

output/
    index.html
    blog/
        index.html

Notice that Bloggers.elm gets skipped as it doesn’t expose a top-level view function.

The way this script works is pretty simple really. It uses a script to convert the output of the view functions to strings, and then through node, we extract the value from those ports to write to files with the same name as the module. This is why Blog.Index gets turned into blog/index.html.

Some background

Elm’s virtual dom is based on an implementation of a virtual dom simply called virtual-dom. How many times can we say virtual-dom in a sentence and have it still make sense?

This virtual-dom implementation has some cool libraries based around it, including one called vdom-to-html. What this does is render the virtual dom to HTML, either on the client or one the server. It just takes a Node, then returns the HTML version of that Node as a string. In Elm, the type signature for that function just looks like this

toHtml : VirtualDom.Node -> String

Right off the bat, that will work with the majority of Elm views. For example,

view =
    div
        []
        [ text "Hello world!" ]

htmlVersion =
    toHtml view

generates

<div>Hello world!</div>

and that’s it! Now we have static site generation in Elm!

If you want check this out, go to elm-static-site and start using it! There’s plenty of ways to improve it, so if you want to ask just ask me at @eeue56 on twitter or eeue56 on Slack!


Noah Hall
@eeue56
Engineer at NoRedInk

Data Structures in Elm

$
0
0

When I first started learning Elm, I had very little sense of what data structures were available to me, and which of these data structures I should actually use. Do I use an Array or a List? Should I use a Set if I’m also going to be mapping over it? What’s the difference between an Object in JavaScript and a Record in Elm? Why does the compiler tell me there’s a problem with Dict.Dict, when I’m definitely using a Set? The distinctions and best use cases of each structure become more clear with practice, but hopefully also through reading this blog post.

Records

What are Records and How Do We Make One

Records (semantically named) are frequently employed to record state or to model information. They employ key-value pairs, with (likely-familar) dot notation for accessing values.

To create a record, you can write out the desired key-value pairs: { name = "Wuthering Heights", author = "Bronte", readByMe = True }. (Note that notation here may be backwards from what you expect. Type signatures, in Elm, use a colon. Assignment uses an equals sign)

If we are planning to create more than one record with the same fields on it, we may want to create a type alias that will describe the shape of these records.


type alias LibraryEntry =
    { name : String
    , author : String
    , readByMe : Bool
    }

Creating a record with these fields looks like this:


wutheringHeights =
    { name = "Wuthering Heights"
    , author = "Bronte"
    , readByMe = True
    }

Now we can use LibraryEntry in type signatures, rather than writing out all three requisite fields. We can also use the LibraryEntry as a constructor. LibraryEntry takes three arguments, in order: name, author, and readByMe. It will hand us back a record of the form we desire.


wutheringHeights =
    LibraryEntry "Wuthering Heights" "Bronte" True


diaspora =
    LibraryEntry "Diaspora" "Egan" False

As you can imagine, this pattern comes in very handy when there is a great deal of data to model, or when there are many records that you’d like to create that should all share the same type.

Getting and Setting on Records

Now we have wutheringHeights and diaspora, representing books in our library. But wait! I actually read Diaspora by Egan two weeks ago. We need to update the readByMe field.


{-

Updating records follows this pattern:

newRecordName =
    { oldRecordName | fieldName = newFieldValue }

-}

newDiaspora =
    { diaspora | readByMe = True }

Note that diaspora is unchanged by this operation: diaspora.readByMe == False and newDiaspora.readByMe == True.

Dot notation is not the only way we can access a value on a record. For every field name on a record, we are given a corresponding accessor function named after the field. For example: .readByMe diaspora == False and .readByMe newDiaspora == True.

You will never have to write (\book -> book.name); you can use .name newDiaspora without defining anything additional.

Extensability and Polymorphism

The first iteration of the Elm record was based heavily on (‘Extensible records with scoped labels’ by Daan Leijen), and many features of the Elm record reflect Leijen’s work. The paper is well worth a read for those interested in modeling data. However, the paper’s titular descriptors–“extensible”, “with scoped labels”–no longer apply to Elm records.

Extensibility (which was paired with restrictability) of a record value meant that a record could have fields added to it and removed from it: e.g., { wutheringHeights - name } == { author = Bronte, readByMe = True }. In practice, developers did not find this feature useful, and extensibility of record values was removed.

The removal of the extensibility of records also meant the end to scoped labels (labels of the same name pointing to different values on the same record, e.g. `{ diaspora + name | name {- This type alias uses "a" to say that a LibraryEntry is any record with name, author, and readByMe fields. -} type alias LibraryEntry a = { a | name : String , author : String , readByMe : Bool }

{- This type alias says that it is a LibraryEntry with exactly one more property–isbn. -} type alias LibraryEntryWithISBN = LibraryEntry { isbn : String }

wutheringHeightsWithISBN : LibraryEntryWithISBN wutheringHeightsWithISBN = { name = “Wuthering Heights” , author = “Bronte” , readByMe = True , isbn = “1853260010” }

Records in Elm remain polymorphic, meaning they can be comprised of fields with all sorts of different types. While I’ve only used Strings and Bools in the library example above, records can hold any values. The following is reasonable, for instance:


type alias Widget entry =
    { text : entry -> Html
    , attributes : entry -> Html
    , getState : entry -> Bool
    }

Records are also transitive with respect to their fields. That is, defining:


oneBeforeTwo =
    { one = 1
    , two = 2
    }


twoBeforeOne =
    { two = 2
    , one = 1
    }

We will find that oneBeforeTwo and twoBeforeOne are equal, even though the fields are declared in different orders. If, though, in twoBeforeOne our field values were strings instead of numbers, we’d find that the compiler would not let us make the comparison at all: we’d be causing a type mismatch.

This is because Records in Elm are structurally typed: the use of records is dependent on the type of their contents. We can’t just compare Records with each other because they’re both Records–we can compare Records if all their fields have the same types.

When we create functions consuming Records, they can be as general or as specific as we please. I can write a getName function that takes any Record with a name field, or I can write a displayHeader function that requires name, author, publishDate, editDate, yourFirstBorn, and maybe a list of related titles too, just because.


getSentenceForLibraryEntry : LibraryEntry -> String
getSentenceForLibraryEntry {name, author} = -- note that Elm gives us clean desctructuring
    author ++ " wrote " ++ name ++ "."


getSentenceForLibraryEntry : LibraryEntry -> String
getSentenceForLibraryEntry libraryEntry =
    libraryEntry.author ++ " wrote " ++ libraryEntry.name ++ "."

Tuples

The Basics of Tuples

Tuples are similar to records, except that rather than having named fields, the meaning of the fields comes from their order. Plus, Tuples are comparable (which, for our purposes, is most relevant when talking about Sets and Dicts, so we’re going to leave it for now).

Let’s talk middle school math for a hot second. Suppose I want to graph some points on to the XY plane. I could create a point without Tuples:


type alias Point =
    { x : Float
    , y : Float
    }

aPoint =
    Point 1 0

So that’s fine. But we have a well-known pattern for expressing XY coordinates. Do we really want to have to say Point or type out { x = 1, y = 0 }every time we want to make a point? With tuples, we don’t need to.

Tuples are denoted with () and contain comma-separated values. So writing the coordinate (1,0) in an Elm Tuple just looks like (1,0).

To access the values of a Tuple, we are given a couple of helpers. fst (1,0) gives us the first value, 1, and snd (1,0) gives us the second value, 0. So, using aPoint from above, I might have x = fst aPoint and y = snd aPoint.

There’s more than one way to create a tuple.


{- We can write it out explicitly. -}
yeOldeBasicCoord =
    (1, 0, 2)


{- We can also use a prefix operator, `,`, to construct the tuple. -}
yeFahncyCoord =
    (,,) 1 0 2

Destructuring Tuples

Wait–but now there are three values in the tuple… So far we’ve only seen fst and snd for accessing the values in a tuple. Elm’s destructuring is quite powerful, and we can use it to access any value in a tuple. We can write our own function for accessing the third value in a tuple, if we want:


{- "_" here is used to indicate that we don't use the first two values from the tuple. -}
z : (Float, Float, Float) -> Float
z (_, _, thirdVal) =
    thirdVal

We can write out a function that uses more than one value from a tuple:


{- Note the type signature here! Tuples' values don't all need to be of the same type. -}
formatNameAndAge : (String, String, Int) -> String
formatNameAndAge (firstName, lastName, age) =
    firstName ++ " " ++ lastName ++ " (" ++ (toString age) ++ ")"

So suppose we wanted to refer to author Neil Gaiman in this text: we might say formatNameAndAge ("Neil", "Gaiman", 55), or “Neil Gaiman (55)”.

Lists

Square Brackets and What’s Inside Them

In Elm, [] signifies an empty list. Lists are just linked lists (ordered collections where each node of the collection points to the subsequent node, if there is one), where all the values must be same type. You get head and tail, just like you’d expect.

head gives you a Maybe with the value of the first element. tail gives you the rest of the List. Want to add an element? Cons :: is an infix operator that adds to the front of the List. You can concat, append, or intersperse. Or partition or unzip or pretty much whatever else you can think of to do. My point: yay lists!

List Operations in Practice with Elm Html

Lists are vital for something I do a lot of: writing HTML in Elm through the elm-html package. The elm-html DSL is very clean: the functions generating HTML nodes are named familiarly (e.g., div, span) and take two lists as arguments, which leads to very readable code.

When using the node-generating Html functions, the first argument is a list of attributes on that node (we also have functions like id and class for producing these attributes). The second argument is a list of Html–that is, a list of children.

So, something like this is reasonable:


renderHello : Html
renderHello =
    div
        [ class "hello" ]    -- We could also throw in a click listener here, or add our own custom attribute or property
        [ text "Hello world!"
        , span [] [ text "oh look I'm a child of the div" ]
        ]

Reasonable, but boring. Let’s go back to our library.


type alias LibraryEntry =
    { name : String
    , author : String
    , readByMe : Bool
    }


wutheringHeights =
    LibraryEntry "Wuthering Heights" "Bronte" True


diaspora =
    LibraryEntry "Diaspora" "Egan" False

We need more books. And do we really need to have wutheringHeights and diaspora floating around cluttering the namespace? Let’s make a list of our books instead of naming each individually.


myListOfBooks : List LibraryEntry
myListOfBooks =
    [ LibraryEntry "Wuthering Heights" "Bronte" True
    , LibraryEntry "Diaspora" "Egan" True
    , LibraryEntry "A Handmaid's Tale" "Atwood" True
    , LibraryEntry "Doors of Stone" "Rothfuss" False
    , LibraryEntry "The Taming of the Shrew" "Shakespeare" False
    , LibraryEntry "The Outsiders" "Hinton" True
    , LibraryEntry "A Scanner Darkly" "Dick" True
    ]

Let’s create a function for displaying a single book:


viewBook : LibraryEntry -> Html
viewBook {name, author, readByMe} =
    div
        [ classList [ ("is-unread", readByMe) ] ] {- classList takes List (String, Bool),
                                                     where each tuple represents a className and whether to apply it -}
        [ div [ class "book-name" ] [ text name ]
        , div [ class "book-author" ] [ text author ]
        ]

Great! Now how should we display our bookList?

Our bookList is just a list of LibraryEntry’s, and the second argument to the containing element needs to be a List of Html… We just need to map from data to Html directly.

My favorite style is to use the |> operator. This style makes it very easy to follow what data is important and most complex. This version of viewBookList emphasizes that the contents of the “book-collection-container” div is just a transformation of bookList.


viewBookList : List (LibraryEntry) -> Html
viewBookList bookList =
    bookList
        |> List.map viewBook
        |> div [ class "book-collection-container" ]

Here’s another version of the same function. This version mirrors the familiar shape of HTML more closely. Some disadvantages: if we decide later to add another mapping or operation on the bookList, it will be harder to add if using the style below (above, it’s just adding a line: List.filter .readByMe, for instance).


viewBookList : List (LibraryEntry) -> Html
viewBookList bookList =
    div
        [ class "book-collection-container" ]
        (List.map viewBook bookList)

Having our data stored in a List means that it is very, very easy to map from data -> view. And immutability means that we can do whatever we want to our Lists without worrying about causing unintended changes somewhere.

The ease with which Lists can be manipulated can be helpful with odd-shaped data. For instance, suppose I want to combine my bookList with someone else’s bookList.


viewTwoBookList : List (LibraryEntry) -> List (LibraryEntry) -> Html
viewTwoBookList bookList willsBookList =
    bookList
        |> List.append willsBookList
        |> List.map viewBook
        |> div [ class "book-collection-container" ]

I could also create my Lists of Html with viewBook, and then append the Lists.

Suppose I stole one of Will’s books. I don’t want to store it in myListOfBooks, because it’s not mine, but I do want this one book to show up in my Html. I can add this book I’ve stolen to the front of my bookList, wherever I’m calling viewBookList:


viewAllBooks =
    let
        theBookIStole =
            LibraryEntry "Axiomatic" "Egan" True

        completeBookList =
            theBookIStole :: myListOfBooks
            {- This infix operator (::) is pronounced "cons".
               That is what I did to steal his book. ;)
            -}
    in
        viewBookList completeBookList

One core function worth mentioning is indexedMap. Coming from JavaScript, this may feel like it’s going to be extremely important. In JS, I found that I often used indices as a key way to access my data (this is a joke. Key? Get it? Because Arrays are just fancy Objects?). Depending on what you’re doing, this function might be helpful to you too.

Maybe you want to zebra-stripe your book entries in a silly way (no nth-child(2n+1) selectors for you).


viewBook : Int -> LibraryEntry -> Html
viewBook index {name, author, readByMe} =
    div
        [ classList
            [ ("is-unread", readByMe)
            , ("is-even", ((index + 1) % 2 == 0))
            ]
        ]
        [ div [ class "book-name" ] [ text name ]
        , div [ class "book-author" ] [ text author ]
        ]

As you may notice from the modular arithmetic going on in this example, Lists in Elm are 0-indexed.

Other functions to be aware of are concatMap and filterMap. Both are semantically named. concatMap applies a function to each element in a List and concatenates the resultant Lists. filterMap applies a function to each element in a List, and keeps the successes.

As you start doing all kinds of fancy nonsense, take a look at elm-list-extra.

Arrays

Arrays are ordered collections, where each item in the collection must be of the same type. In Elm, [] doesn’t correspond to arrays ([] is an empty list, discussed above). To create an empty array, use Array.empty.

To create an Array of a given length with a specific default element, do: Array.repeat 9 "Tessa". This will give you an Array filled with 9 “Tessa”’s. Some might say this is too many. A third option is to use Array.initialize, whose first argument is the desired length of the array. Array.initialize’s second argument is a function expression operating on the index; the result of this operation is the value of the array at that index.


arrayOfTessas =
    Array.initialize 9 (\index -> "Tessa" ++ (toString index))

Then arrayOfTessas will be an Array that in JavaScript-land looks like: ["Tessa0","Tessa1","Tessa2","Tessa3","Tessa4","Tessa5","Tessa6","Tessa7","Tessa8"].

Okay, sweet. But obviously “Tessa3” is my favorite, and the one that I want. How do I get it?

In JavaScript, you get a value back for any index or key you ask for in an array, since an array is just an object. (In JavaScript, arr["Something I want"] tends to be undefined, but what can you do.)

In Elm, accessing a value at an index that doesn’t exist will fail explicitly–there’s no question of whether you got undefined because you were doing something marvelous with your array like [0,1,undefined,"Bwahaha that's going to confuse me later"]. Instead, in Elm, you will get back Nothing if there was nothing at that index (this is a Maybe).

Suppose I have an array comprised of my LibraryEntrys from before.


myBooks : Array LibraryEntry
myBooks =
    Array.fromList [ wutheringHeights, diaspora ]
    {- For convenience, we're using Array.fromList to convert this list of library entries into an array. -}

Suppose I try to get the value at the second index. This is going to fail, since arrays’ indexing starts at 0, and the length of the array is 2. But I try it anyway: myBooksSecondIndex = Array.get 2 myBooks. So what’s myBooksSecondIndex? It’s Nothing.

So maybe you’re thinking that there are situations where you might have Nothing in an Array. That’s fine too.


maybeMyBooks : Array (Maybe LibraryEntry) -- Note that the type signature indicates that the elements are Maybes
maybeMyBooks =
    Array.fromList [ Just wutheringHeights, Just diaspora, Nothing ]

Now, if we try to access the second index: myBooksSecondIndex = Array.get 2 maybeMyBooks we have myBooksSecondIndex == Just Nothing. This is unlikely to be a useful way of structuring things, and I’d encourage you to stahp, but it does illustrate the clarity that Elm brings to a familiar JavaScript problem: is that undefined because it’s supposed to be, or because I failed to set a value?

Speaking of setting values: we can push a value to the end of the array, append an Array to another to create a new array, or we can set a value at an index. Setting values is a bit different than in JS.

In JavaScript, I can do something like this:


var aRiddle = "What's the longest word in the English language?"

var myAnswerArray = []

// Hmmmmmmmmmmmm

myAnswerArray.push("S")
myAnswerArray[5279] = "S"

console.log(myAnswerArray.length + " feet") // => "5280 feet"

var myAnswer = "Smiles! There's a mile in the middle!"

In Elm, not so much.


myAnswerArray =
    Array.fromList ["S"]
        |> Array.set 5279 "S"

Array.set will not update the Array it’s passed if the given index is out of bounds. We tried to set “S” as a value at index 5279, but that’s way out of bounds, so this set will just not occur. Array.toList myAnswerArray is just ["S"]. No mile in the middle here!

Array Implementation

Arrays are implemented as relaxed-radix balanced trees. They’re an extension to vectors (as used in Clojure and Scala), but with optimized concatenation. Effectively, even though they’re immutable, they have constant time getting and setting. Obviously, this is magic (read: math). To read the math and learn the magic: Bagwell and Rompf’s “RRB-Trees: Efficient Immutable Vectors”

To read more about Elm’s implementation, read this blog announcement.

As cool as the math behind arrays is, and as familiar as they might seem, we tend not to use them. Currently, there are reliability issues with arrays and many Elm developers tend toward using lists instead. But fear not! Array is being rewritten, in pure Elm instead of Native. Be excited.

Sets

What are sets, and when should you use them?

Elm’s sets are a very valuable data structure, and one that’s worth knowing well. All the basic functionality of sets will be familiar from middle and high school math (also, college level, if you took Analysis or Real Analysis or Abstract Algebra or Logic or Philosophy). A set is a collection of unique items; in Elm, these items must be comparables (ints, floats, chars, strings, lists, or tuples) and of the same type.

So when might it make sense to use a set? Whenever you find yourself doing a lot of filtering to check for List entries’ inclusion in other Lists, you should think about using Sets instead. Is your collection of elements closed? Is your collection of elements a subset of another collection? Should the elements of your collection be unique? Here too you may want to use a Set instead of a List.

In deciding to use a set rather than another data structure, there’s one other important consideration: are the elements of your collection comparable? Sets can only contain comparable values. So our collection of sweet library books? Not going to work as a set.

Using a Set

Suppose that our LibraryEntrys had unique ids. We could store a string id in a set.


type alias LibraryEntry =
    { id : LibraryEntryId
    , name : String
    , author : String
    , readByMe : Bool
    }

type alias LibraryEntryId =
    String

With these ids, we can build a library basket (because library baskets are definitely a thing don’t worry about it there are totally baskets) and a collection of checked out books.


model =
    { yourLibraryBasket = Set.empty -- Set.empty constructs an empty set
    , checkedOutBooks = Set.empty
    }

Now, checking to see if a book (a LibraryEntry with, say, id "10") is checked out is simple: Set.member "10" model.checkedOutBooks returns a boolean describing whether or not “10” is an element in model.checkedOutBooks. Is the book already in our basket? Set.member "10" model.yourLibraryBasket will tell us.

Implementation

Sets are actually just a wrapper around Elm’s Dicts, which are implemented with a red black tree. Be aware that as a result of this implementation, == is not reliable (with Sets or Dicts). The order in which elements are added to a RB-tree impacts the form the tree takes, which makes equality a sticky concept.

Let’s walk through an example.


set1 = Set.fromList ([1,2,3,4])

set2 = Set.fromList ([1,2,3,4])

set3 = Set.fromList ([4,3,2,1])

We have three sets, each with the same values: 1, 2, 3, and 4. The only difference between the Sets is in the order in which these values are added. For these sets, does the root node have value 2 or 3? Who knows?


listVersionsAreOrdered =
    Set.toList set1 == Set.toList set2 && Set.toList set2 == Set.toList set3 -- True

setsCreatedInSameOrderAreEqual =
    set1 == set2 -- True

uhOhNotEqual =
    set2 == set3 -- False

Moral: don’t ask whether sets are equal, because what you’re asking is “are these two trees made out of the same values && were these trees balanced in the same way?” It’s worth noting that this warning about equality might not always apply. Current plans for Elm include changing the equality behavior for sets and dicts to not be dependent on the balancing of an RB-tree. For now, the documentation provides a warning.

Dicts

What is a Dictionary?

A dict maps unique keys to values. There’s a fair amount of overlap in the use cases for dictionaries and sets, but dictionaries allow for storing a bit more information. Since values are indexed by unique keys, we can try to Dict.get 'some comparable' Dict.empty, which may give us Nothing.

Our Library Has a Dictionary

What’s one book libraries should definitely have? That’s right, dictionaries.

Our library book loaning system from the set example might work better as a dict. With a dict, we can tie checked out ids to library entries directly.


type alias LibraryEntry =
    { id : LibraryEntryId
    , name : String
    , author : String
    , readByMe : Bool
    }


type alias LibraryEntryId =
    String


model =
    { yourLibraryBasket = Dict.empty -- Dict.empty constructs an empty dict
    , checkedOutBooks = Dict.empty
    }


henryV = LibraryEntry "0" "Henry V" "Shakespeare" True

Now, suppose we want to add Henry V to our library basket: Dict.insert henryV.id henryV model.yourLibraryBasket will give us back a dictionary with a key “0” pointing to a LibraryEntry describing the play Henry V. This can make it very easy to work with henryV.

Suppose we just want to display books that are in our basket. When we were using a set to hold the ids of our books, this would have required some filtering/checking every book id to see if it was in the set of basket ids or not. With a dict, we can do something similar, (Dict.member "0" model.yourLibraryBasket == True), but we can also just grab all of the values in our Dict:


type alias Model =
    { yourLibraryBasket : Dict LibraryEntryId LibraryEntry -- Note the type signature here
    , checkedOutBooks : Dict LibraryEntryId LibraryEntry
    }


model =
    -- Dict.singleton creates a dictionary with 1 key-value pair. henryV is this value.
    Model (Dict.singleton henryV.id henryV) Dict.empty


viewBook : LibraryEntry -> Html
viewBook {name, author, readByMe} =
    div
        [ classList [ ("is-unread", readByMe) ] ] {- classList takes List (String, Bool),
                                                     where each tuple represents a className and whether to apply it -}
        [ div [ class "book-name" ] [ text name ]
        , div [ class "book-author" ] [ text author ]
        ]


viewBookList : List (LibraryEntry) -> Html
viewBookList bookList =
    bookList
        |> List.map viewBook
        |> div [ class "book-collection-container" ]



viewLibraryBasket : Model -> Html
viewLibraryBasket model =
    let
       yourBasket =
            model.yourLibraryBasket
                |> Dict.values -- This gets us just the values in a list
                |> viewBookList
    in
        div
            [ class "basket" ]
            [ h3 [] [ text "Your Basket" ]
            , yourBasket
            ]

Now, how do we remove books from our basket? The semantic Dict.remove "0" model.yourLibraryBasket. Let’s write out a function that will either remove books from our basket or add them to our basket, on a case-by-case basis.


type Action
    = RemoveFromBasket LibraryEntryId
    | AddToBasket LibraryEntry


update : Action -> Model -> Model
update action model =
    case action of
        RemoveFromBasket libraryEntryId ->
            { model | yourLibraryBasket = Dict.remove libraryEntryId model.yourLibraryBasket }

        AddToBasket libraryEntry ->
            { model | yourLibraryBasket = Dict.insert libraryEntry.id libraryEntry model.yourLibraryBasket }

At this point, you might be seeing the Elm Architecture beginning to emerge. We’re not going to wire everything together here, since our focus is on the data structures rather than the architecture. As an exercise, try building an mini-application out of what we’ve begun here.


Tessa Kelly
@t_kelly9
Engineer at NoRedInk

Pub/Sub in 30 Lines of Elixir

$
0
0

All programming languages give you some primitives to use as starting points in your own creations. For example, a language might provide arrays, hashes, and objects that you turn into a system representing students, teachers, and the work that passes between them.

Some languages go a little further in what they provide. For example, several languages have primitives that represent code in that language. This opens the doors to using metaprogramming to remake the language into exactly what you need it to be to best solve your problems. The various Lisps are perfect examples of this.

Now Elixir includes process primitives. These processes are actually inherited from Erlang, which Elixir is built on top of.

The name is a little confusing because you can spin up a “process” from most any language. That’s not the same thing. In most languages, the word process refers to a thing created and managed by your operating system. These are completely separate programs. Operating systems are heavily optimized for creating these programs but they are still pretty heavyweight items. Elixir processes are much smaller.

You might be wondering if they are more like threads? Not really. Threads are still too big and they deal in terms of shared memory. Shared memory usually involves some kind of locking and this quickly complicates their usage. Elixir processes share nothing.

Finally, communication is baked in. Elixir processes can send messages to each other.

In many ways, Elixir processes are closer to objects in Object Oriented languages. They encapsulate data and it’s the interactions among the processes that make up what a system does. The differences are that processes can act independently of each other, even in parallel, and you are forced to send messages to them.

Let’s look at an example. The following Elixir code launches two processes and has the second one send a message to the first. We’ll examine it in chunks:

IO.puts "#{inspect self} is a process launching other processes..."

# ...

This first bit just prints a statement indicating what the main process is about to do. By main process, I just mean the one that started running your code, as opposed to the other processes we are about to launch. inspect self just shows a unique identifier for each process, so we can tell them apart in the output.

# ...

receiver = spawn(fn ->
  IO.puts "#{inspect self} is a process listening for messages..."
  receive do
    %{from: from, content: content} ->
      IO.puts "#{inspect self} received this message from #{inspect from}:  " 
              content
  end
end)

# ...

This code creates our first subprocess. That new process also prints output regarding its job and before doing what it said it would: listening for incoming messages. It will print another line when a message arrives.

# ...

spawn(fn ->
  IO.puts "#{inspect self} is a process sending messages..."
  send(receiver, %{from: self, content: "Hello other process!"})
end)

This creates our second subprocess. Like the others, it prints what it will do, then does it. In this case, the job is to deliver a message to the first subprocess.

If we bundle all of that code up in a file called processes.exs, it would look like this:

IO.puts "#{inspect self} is a process launching other processes..."

receiver = spawn(fn ->
  IO.puts "#{inspect self} is a process listening for messages..."
  receive do
    %{from: from, content: content} ->
      IO.puts "#{inspect self} received this message from #{inspect from}:  " 
              content
  end
end)

spawn(fn ->
  IO.puts "#{inspect self} is a process sending messages..."
  send(receiver, %{from: self, content: "Hello other process!"})
end)

There are three key bits in the code:

  • The spawn() function launches a new process. In this case, I have provided anonymous functions for the code I want to run in each one. A PID (Process ID) is returned, which is useful with…
  • The send() function allows one process to send a message to another process. The message can be any data. In this case, I’ve used a Map, which is Elixir’s equivalent to Ruby’s Hash.
  • Code inside of receive do … end allows a process to listen for incoming messages. Inside, various clauses pattern match the incoming messages and provide code for handling them. A listening process will block here until a message arrives that matches one of the patterns or an optional timeout is hit (not shown in this post).

You can see how these pieces interact if you run the code with Elixir 1.2.3:

$ elixir processes.exs
#PID is a process launching other processes...
#PID is a process listening for messages...
#PID is a process sending messages...
#PID received this message from #PID:  Hello other process!

This may not look like much yet, but you can build fully distributed systems using just these tools. Let’s prove it by creating a pub/sub messaging system in 30 lines of code. I’ll keep my processes on the same machine for this example, but the code we build up could be used across the network without changes. Elixir makes that trivial.

A Pub/Sub Server

Elixir already gives us a way to send messages to individual processes as needed. Let’s say we need a Publish-Subscribe Channel though. In that model, interested processes subscribe to the channel and then, when a message comes in, it is forwarded to all current subscribers.

To give an example of how this could be useful, imagine that we have events arriving that represent student’s answers to quiz questions. The main action is to figure out if the student has answered correctly and respond to them. Several subsystems may also be interested in the result though. We may need to update the students mastery scores for the concept that the question covers; we may need to update a teacher’s grade book based on how the student did; we might want to record some general metrics useful in determining the overall difficulty of the question. One way to avoid tightly coupling all of these systems to the evaluation response code, is to allow the latter to publish results that subsystems can subscribe to receive. If we someday wish to track a new metric that we haven’t paid attention to in the past, we can just add another subscriber for the new subsystem.

We need three things to pull this off:

  1. We need to spawn() a PubSubServer process. Its job will be to track who is subscribed and handle incoming messages.
  2. We need a way to send messages to that server process.
  3. We need a way to subscribe to messages coming out of the server process.

We’ll attack the problem in just that manner. Let’s build up some code in a file called pub_sub.exs. First, we spawn() a process:

defmodule PubSubServer do
  def start do
    spawn(__MODULE__, :run, [[ ]])
  end

  # ...
end

You’ve seen spawn() before, but there are two differences in this code. First, I am tucking these functions away in their own module to namespace the API provided. The other difference is that I have not passed spawn() an anonymous function this time, but instead referenced a function using three identifiers. The first is a module name and I used __MODULE__ to fetch the name of the current module. The second is the name of the function to call in that module: run(), which we’ll get to in a bit. Finally, you provide a list of arguments to pass to that function, which in this case is a single argument that is an empty list.

My start() function returns what spawn() does: the PID of the new process. We can use that to communicate with the process. Given that, what would publishing a message look like?

defmodule PubSubServer do
  # ...

  def publish(server, message) do
    send(server, {:publish, message})
  end

  # ...
end

This is a trivial wrapper over send(). Why do that? The caller has the PID and it could just call send() themselves. That’s true, but by wrapping it we remove the need for the caller to know our internal message format. We may choose to change that someday and they won’t even need to know, as long as they keep calling publish(). For now we’ve chosen to make messages simple tuples where the first element is an atom describing the message type. That can be used in pattern matching on the receiving end.

Subscribing is similar, but we’ll do a little more for the user:

defmodule PubSubServer do
  # ...

  def subscribe(server, handler) do
    send(server, {:subscribe, self})
    listen(handler)
  end

  def listen(handler) do
    receive do
      message ->
        handler.(message)
        listen(handler)
    end
  end

  # ...
end

Again, we send() the message on the caller’s behalf. This time we say that self is subscribing, which is just how you get the PID of the current process in Elixir. After the message is sent we also fall into a receive loop to wait for the incoming messages. The caller provides an anonymous function that we’ll call with each message we receive to invoke whatever code they need.

The listen() function might look a little weird if you haven’t seen a “loop” in Elixir before. The language doesn’t really have loops, so functions just recurse to perform some operation repeatedly. Because Elixir has tail call optimization, this process doesn’t add new stack frames and is very efficient.

The outside interface for our code is defined. We just need that run() function with the internal logic we promised in our spawn() call.

defmodule PubSubServer do
  # ...

  def run(subscribers) do
    receive do
      {:publish, message} ->
        Enum.each(subscribers, fn pid -> send(pid, message) end)
        run(subscribers)
      {:subscribe, subscriber} ->
        run([subscriber | subscribers])
    end
  end
end

This code handles the two types of messages that we made up in the previous functions. When :publish comes in, we forward the message to our list of subscribers passed into run(). The :subscribe message is how that list grows and it is another slightly tricky Elixir-ism.

I mentioned before that loops are just recursive function calls and we see that again in run(). The interesting new bit is how processes “remember” things.

Elixir’s data structures are immutable, so any “change” actually makes a new structure that reflects the differences. That might leave you wondering how we keep track of which of these structures is current?

The answer is that we pass them as arguments to the recursive calls. This means that each pass through the loop can learn from what happened in previous passes. If you recall back to the start() function, I passed an empty list to the initial call of run(). Each time a :subscribe message comes in, we push the new listener onto the front of the current list as we hand it down to the next call of run(). When a :publish message is received, subscribers will have accumulated the listeners from all previous times through the loop.

That’s it. We’ve built a pub/sub system. All we have to do is prove it.

When Everything Happens At Once

My first naive thought was to show some example code like this:

server = PubSubServer.start
listener_count = 10

Stream.repeatedly(fn ->
  spawn(fn ->
    PubSubServer.subscribe(server, fn message ->
      IO.puts "#{inspect self} received:  #{message}"
    end)
  end)
end) |> Enum.take(listener_count)

PubSubServer.publish(server, "Hello everyone!")

This code fires up our PubSubServer, creates a bunch of subscribers, and publishes a message through the system. The code in the middle may look a bit odd because Elixir doesn’t have Ruby’s 10.times do … end iterator. I recreated it by building an infinite Stream that just calls an anonymous function repeatedly() and then take()ing ten values from it.

But does this code work? Yes and no, but it’s going to look more like no:

$ elixir pub_sub.exs
$

The surprised look on my face right now reminds me that I forget to mention one very important detail when I started this discussion: Elixir processes are fully parallel. When you call spawn() it sets up the new process, but control returns to the caller as soon as it has a PID to give you. The calling code doesn’t actually wait for anything to happen in that child process before it goes back to executing instructions in the parent process.

Given that, I introduced two separate timing bugs in the example code above! Can you puzzle them out?

The first issue reveals itself if we add some print statements to check the timing:

defmodule PubSubServer do
  # ...

  def run(subscribers) do
    receive do
      {:publish, message} ->
        IO.puts "Publishing..."
        Enum.each(subscribers, fn pid -> send(pid, message) end)
        run(subscribers)
      {:subscribe, subscriber} ->
        IO.puts "Subscribing..."
        run([subscriber | subscribers])
    end
  end
end

Here’s what they tell us:

$ elixir pub_sub.exs
Publishing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...

Kind of backwards from what we were aiming for, eh? As a refresher, here’s the problematic code:

# ...

Stream.repeatedly(fn ->
  spawn(fn ->
    PubSubServer.subscribe(server, fn message ->
      IO.puts "#{inspect self} received:  #{message}"
    end)
  end)
end) |> Enum.take(listener_count)

PubSubServer.publish(server, "Hello everyone!")

The calls to spawn() are triggered first, but the main process keeps going as those processes start running. It finishes the call to publish() before the processes can get subscribed and that message is passed on to zero processes. That’s our first problem.

A clumsy fix is just to slow the main process down a touch before it publish()es:

# ...

# give the subscribers time to get rolling
:timer.sleep(1_000)

PubSubServer.publish(server, "Hello everyone!")

OK, the forced sleep is gross, but does it work? Again, it’s an unsatisfying yes and no:

$ elixir pub_sub.exs
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Publishing...

It’s obvious that fixed the ordering problem, but we still don’t see the output from our forwarded messages. This is our second problem. It’s really just the first problem again, but the interaction is less obvious to me.

After publish() is called the message starts speeding through the PubSubServer and the subscriber processes. Again though, the main process carries on. Well, the opposite actually. Since the call to publish() is the last line the main process executes, it exits right after and shuts down the whole virtual machine before the subscriber processes complete their work.

One more sleep will show us that PubSubServer has worked fine all along. Here’s the added line:

# ...

PubSubServer.publish(server, "Hello everyone!")

# give the messages time to be written
:timer.sleep(1_000)

And here’s the output we’ve been waiting for:

$ elixir pub_sub.exs
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Subscribing...
Publishing...
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!

I know what you’re thinking: can you do it without the calls to sleep()? Why, yes I can. Stick around.

There’s another way around our use of the second sleep(). If I remove the extra calls to puts() and that final sleep(), we can still make it work:

$ elixir --no-halt pub_sub.exs
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
^C
BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
       (v)ersion (k)ill (D)b-tables (d)istribution
q

The difference here is the usage of Elixir’s --no-halt flag. Sometimes people write scripts that spin up various processes to do stuff and they want to wait on that work even though the main process is finished. That’s why this flag exists. It tells Elixir to skip the step of shutting everything down when it falls off the end of the main script. The virtual machine just keeps running. You can see that I manually stopped it here by typing control-c, q, and return.

That’s still kind of cheating though, right? Let’s roll our sleeves up and really solve these timing issues.

Knowing What a Process Is Thinking

In order to fix the example code, we need two pieces of knowledge that we don’t currently have. First, we need to know when the PubSubServer has ten subscribers. We want to publish the message after that point, so we need to wait on it to happen.

The second piece of knowledge we need is similar. We need to know when the subscribers have finished acknowledging that they received the messages from PubSubServer. That’s the point when it’s safe for us to shutdown.

In other words, we have two cases of needing to know what a process is thinking. Which leads us to a great rule of thumb: if you need to know what an Elixir process is thinking, it needs to tell you. We can use this telling strategy to fix up our code.

To gain the first piece of knowledge, we do need to thread a callback through two functions of PubSubServer. Here are the changes:

defmodule PubSubServer do
  def start(subscriber_callback \\ nil) do
    spawn(__MODULE__, :run, [[ ], subscriber_callback])
  end

  # ...

  def run(subscribers, subscriber_callback) do
    receive do
      {:publish, message} ->
        Enum.each(subscribers, fn pid -> send(pid, message) end)
        run(subscribers, subscriber_callback)
      {:subscribe, subscriber} ->
        if subscriber_callback, do: subscriber_callback.(subscriber)
        run([subscriber | subscribers], subscriber_callback)
    end
  end
end

The difference here is that we now accept an optional subscriber_callback when the server is start()ed. We pass that callback down to run() and add it to what we are keeping track of while recursing. Then, if a callback is provided, we call the function whenever a new process subscribes.

Now we can half-fix the example code:

# add a callback that sends us a message when someone subscribes
main = self
server = PubSubServer.start(fn subscriber ->
  send(main, {:subscriber, subscriber})
end)
listener_count = 10

Stream.repeatedly(fn ->
  spawn(fn ->
    PubSubServer.subscribe(server, fn message ->
      IO.puts "#{inspect self} received:  #{message}"
    end)
  end)
end) |> Enum.take(listener_count)

# replace `sleep()` with listening for `listener_count` subscribers
Stream.repeatedly(fn ->
  receive do
    {:subscriber, _} -> true
  end
end) |> Enum.take(listener_count)

PubSubServer.publish(server, "Hello everyone!")

The comments point out the two changes made: we added a callback that notifies main when a process subscribes and we now listen for all of the subscribers to join before we publish() a message. That removes the need to sleep, though we still need the --no-halt flag:

$ elixir --no-halt pub_sub.exs
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
^C
BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
       (v)ersion (k)ill (D)b-tables (d)istribution
q

Doing pretty much the same fix one more time finally banishes the timing issues from our code:

main = self
server = PubSubServer.start(fn subscriber ->
  send(main, {:subscriber, subscriber})
end)
listener_count = 10

Stream.repeatedly(fn ->
  spawn(fn ->
    PubSubServer.subscribe(server, fn message ->
      IO.puts "#{inspect self} received:  #{message}"
      send(main, {:written, message})  # notify `main` of the write
    end)
  end)
end) |> Enum.take(listener_count)

Stream.repeatedly(fn ->
  receive do
    {:subscriber, _} -> true
  end
end) |> Enum.take(listener_count)

PubSubServer.publish(server, "Hello everyone!")

# wait for `listener_count` writes before we quit
Stream.repeatedly(fn ->
  receive do
    {:written, _} -> true
  end
end) |> Enum.take(listener_count)

Again, see the comments. They point out the new notification and the final pause for incoming messages.

Our example code finally functions as advertised:

$ elixir pub_sub.exs
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!
#PID received:  Hello everyone!

In Your Own Code

You can read through the final code from this post if you’re curious, but know that it’s just a learning tool. If you really need an Elixir pub/sub system, you should probably borrow the one the Phoenix web framework uses or at least rework my code in terms of GenEvent, an Elixir tool would allow you to use supervisors to restart your process in case of failure.

Having these powerful primitives to build on gives you some choices when writing Elixir applications. In many other languages, the norm would be to add external dependencies to handle pub/sub, queuing, scheduled job execution, and more. You can still take that path in Elixir too. It works as it always has. With Elixir though, you also have the choice of easily meeting your specific need using the tools included with the language. It’s always nice to have options.


James Edward Gray II
@JEG2
Engineer at NoRedInk

Designing Meetups to Build Better Communities

$
0
0

SF Elm Hack Night

Switching from Presentations to Hack Nights

The number of tech meetups in San Francisco is pretty insane. You could probably eat free pizza every night of the week if that was your thing.

Despite the quantity of meetups here, they almost all have the same structure. Tens of people show up, everyone has pizza and beer, a person gives a presentation for 30 or 60 minutes, people ask questions or flex at the speaker, people mill about for a bit, and then everyone goes home. Okay, but is that actually a nice way to build a community? I mean, sometimes the talk is boring. Sometimes the talk is interesting, but I only really understood 50% of it. Maybe I want to ask someone, but it is hard to start talking with strangers. Often I end up talking to between zero and one new person.

Early on with the Elm meetup in SF, I switched to a hack night style. I wanted the meetup to be more inclusive and beginner friendly. I wanted to see what sort of cool things everyone else was working on. Basically, I wanted to build a fun community where people learn and make cool stuff together!

Now that there are Elm meetups popping up in cities all over the world, I figured it was a good time to share my thinking on this!

What is a Hack Night?

We structure the Elm meetups in San Francisco like this:

  • Sometimes spend 10 or 15 minutes up front showing something cool. Maybe that is me previewing improved error messages or Richard demoing elm-css or Aaron showing elm-format. This is more of a show-and-tell about making something cool. If there is nothing to show, we skip this.

  • Pairing and collaboration is kind of the whole point of these meetups, so we have a couple minutes where folks can briefly share what they will be working on. This does two important things. First, it makes it clear to everyone that this meetup and this community is about building cool and useful stuff. (Not all functional language communities are this way!) Second, it gives new folks an easy way to find some form of mentorship. Maybe that just means asking a few questions at the meetup, but hopefully it means finding collaborators on interesting projects!

  • From there we work together until the place clears out hours later!

One thing to notice is that “the spirit of Elm” is embodied by this structure. Elm is about making great projects and having fun programming. It is about the joy of learning. It is about bouncing ideas off others to find better answers. So the Hack Night format is a great cultural fit for us.

Not all parts of the presenter/audience meetup style is gone though. We definitely still have free pizza and beer!

SF Elm Hack Night

Before and After

I did maybe three meetups in the traditional presenter/audience style before switching, and it was amazing to see how the attendees changed almost immediately.

In one of the presenter/audience meetups, the topic was “The Taxonomy of FRP” and the audience was a mix of PhDs, authors of reactive libraries, and maybe a few folks curious what this Elm thing is. Most people had backgrounds in languages like Haskell and Scala. Now this was a fun event, but in a way, this was actively harmful for Elm. I was helping experts get even farther from mainstream programming! This is not how typed functional languages are going to break out of their niche.

As soon as I switched to the hack night style, the attendees totally changed. It became primarily folks who write JavaScript for work. (We also get a reliable subset of folks using backend languages like Ruby, Clojure, Erlang, etc. who are trying to get into frontend programming without going crazy.) More importantly, a lot of these folks are very new to Elm. Often people will come planning to get things installed while at the hack night. This is great! We end up with a healthy mix of new and experienced people, so everyone is learning from each other. Beginners gain confidence quicker by asking questions and working with others. Package authors stay grounded in reality, seeing how their API works for the real life humans they are trying to help. Core folks such as myself learn from novices and experts alike, making it possible to make targetted improvements to docs, libraries, etc.

So with hack nights, Elm is just a backdrop for meeting interesting folks and building cool stuff together. I think this was a huge part of how we built a community that is friendly and fun.

Designing Culture

Now I knew I did not want a presenter/audience style for my Elm meetups, but I was pretty clueless about what to do instead until I talked with Cecily Carver about how she ran Dames Making Games in Toronto. She shared ideas about collaboration and community building that were the primary inspiration for the hack night style meetups we ended up with. Thank you again Cecily!

Now, maybe a hack night will work well for your community, but maybe not! The point here is more that the spirit of your project manifests in your meetups. Elm is about everyone making cool stuff, so the meetups are too! Papers We Love is about reading interesting academic computer science papers, so maybe only the community building aspects of a hack night are interesting. For some projects, maybe nothing really translates at all! The real point here is that: even if you do not think about it, your meetups help define your community and its culture. If you are concious of this, you will end up with a healthier community and culture.

That said, it is super easy to be blind to your own failings. Sometimes people within a community may see themselves a certain way, but the meetups tell a different story. Maybe its like that for us! The only way to know is to come to one of the Elm meetups at the NoRedInk offices in SF and check it out. Beginners are welcome! (Feedback on how to do better is welcome too!)


Evan Czaplicki
@czaplic
Engineer at NoRedInk

The Most Object-Oriented Language

$
0
0

I’ve been listening to the Elixir Fountain podcast. It’s a pretty great source of interviews with people who have played some part in the rise of the Elixir programming language.

Episode 19 was particularly interesting to me. It’s an interview with Joe Armstrong, co-creator of Erlang. The discussion includes a lot of details about the platform Elixir is built on top of.

There’s one moment in the episode that surprised me. Joe makes a comment about Erlang being “the most object-oriented language.” The same can be said of Elixir.

Wait, what?

Does Elixir even have objects? No. Isn’t Elixir a functional programming language? Yes.

When I first picked up Elixir, I heard some developers comment that it was an easy transition from Ruby because it was similar in some ways. Early on though, I decided I didn’t agree with that. The syntax was Ruby inspired, sure, but with immutability, pattern matching, and recursion everywhere, it felt like a significantly different beast to me.

Then I started to dig into processes more. As I got deeper, Joe’s comment from the podcast culminated in a significant “Ah ha” moment for me. Let’s see if we can gain a deeper understanding of what he meant.

What Is Object-Orientation?

This is a pretty challenging question for developers. We don’t often seem to agree on exactly how to define this style of programming.

The first book I ever read about object-oriented programming explained it from the point of view of C++ and it drilled into me that it was a combination of three traits: encapsulation, inheritance, and polymorphism. That definition has drawn increasing criticism over time.

A likely better source of definitions for the term is the man who coined it, Dr. Alan Kay. There have been attempts to catalog what he has said about object-orientation. My favorite definitions come from an email he wrote to answer this question in 2003. It talks about his original conception, which includes this description:

I thought of objects being like biological cells and/or individual computers on a network, only able to communicate with messages (so messaging came at the very beginning – it took a while to see how to do messaging in a programming language efficiently enough to be useful).

The email culminates in this definition:

OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things.

You may have noticed that these definitions never mention things like classes or instances. Wikipedia talks a lot about such things when defining object-oriented programming but I view that more as a description of where a lot of languages have gone with the concept. The heart of object-oriented programming doesn’t seem to require such bits, at least according to the man who named the style.

OK, But Where Are the “Objects?”

Playing with the semantics of these definitions is fun and all, but it does seem like we would at least need some objects for this idea of object-oriented programming to apply to us. Elixir doesn’t strictly have anything called objects, but let’s use a quick example to look at what it does have.

We’ll create a script that fetches a page off the web, crudely parses out the title, and prints it. This doesn’t have any practical value. The whole point is to give use some code to examine, in Elixir’s normal context.

We’ll begin by creating the project as normal:

$ mix new --sup title_fetcher
* creating README.md
* creating .gitignore
* creating mix.exs
* creating config
* creating config/config.exs
* creating lib
* creating lib/title_fetcher.ex
* creating test
* creating test/test_helper.exs
* creating test/title_fetcher_test.exs

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd title_fetcher
    mix test

Run "mix help" for more commands.

The --sup switch that I added tells Mix to go ahead and give me a standard Supervisor and prepare my application for running by starting said Supervisor. This just gets us going with less effort.

Now, we need a way to fetch the web page so let’s add the HTTPoison library as a dependency of our project. To do that, we open mix.exs and make two small tweaks:

defmodule TitleFetcher.Mixfile do
  # ...

  defp deps do
    [{:httpoison, "~> 0.8.1"}]
  end
end

This first change just adds the dependency.

defmodule TitleFetcher.Mixfile do
  # ...

  def application do
    [applications: [:logger, :httpoison],
     mod: {TitleFetcher, []}]
  end

  # ...
end

The other change was to add :httpoison to the list of applications that Elixir will start up for us. An application in this context means something different from the normal computer term. It helps me to think of it as a reusable component.

With those changes in place, we can ask Mix to fetch our dependencies:

$ mix deps.get
Running dependency resolution
Dependency resolution completed
  certifi: 0.3.0
  hackney: 1.4.10
  httpoison: 0.8.1
  idna: 1.1.0
  mimerl: 1.0.2
  ssl_verify_hostname: 1.0.5
* Getting httpoison (Hex package)
Checking package     (https://s3.amazonaws.com/s3.hex.pm/tarballs/httpoison-0.8.1.tar)
Using locally cached package
* Getting hackney (Hex package)
Checking package     (https://s3.amazonaws.com/s3.hex.pm/tarballs/hackney-1.4.10.tar)
Using locally cached package
...

We’re ready to add our little bit of code. This time we need to open lib/title_fetcher.ex. It contains a commented out line inside the start() function that looks like this:

      # worker(TitleFetcher.Worker, [arg1, arg2, arg3]),

We need to change that to run our code:

      worker(Task, [&fetch_title/0]),

A Task is a built-in tool for wrapping some code in an Elixir process. Here we’ve instructed it to call the fetch_title() function. That’s the only other bit that we need to add:

defmodule TitleFetcher do
  # ...

  defp fetch_title do
    body = HTTPoison.get!("https://www.noredink.com/") |> Map.get(:body)
    Regex.run(~r{<title>([^}, body, capture: :all_but_first)
    |> hd
    |> IO.puts

    System.halt
  end
end

This is the code for what we set out to do. The first pipeline retrieves the contents of NoRedInk’s homepage. The second extracts and prints the title of that page. Then there’s a call to System.halt() to shut everything down.

The end result is probably what you expect to see:

$ mix run --no-halt
Compiled lib/title_fetcher.ex
NoRedInk makes learning grammar fun and easy

The result isn’t what I wanted to show you though. Let’s make this code show us a little about how it did the work. First, we know we had the Task process managing what we were telling it to do, but let’s ask if there were other processes doing stuff:

defmodule TitleFetcher do
  # ...

  defp fetch_title do
    Process.registered |> Enum.sort |> IO.inspect

    body = HTTPoison.get!("https://www.noredink.com/") |> Map.get(:body)
    Regex.run(~r{<title>([^}, body, capture: :all_but_first)
    |> hd
    |> IO.puts

    System.halt
  end
end

There’s just one new line at the beginning of the function. All it does is print out all registered processes for us to inspect. Here’s the updated output:

$ mix run --no-halt
Compiled lib/title_fetcher.ex
[Hex.Registry.ETS, Hex.State, Hex.Supervisor, Logger, Logger.Supervisor,
 Logger.Watcher, Mix.ProjectStack, Mix.State, Mix.Supervisor, Mix.TasksServer,
 TitleFetcher.Supervisor, :application_controller, :code_server,
 :disk_log_server, :disk_log_sup, :elixir_code_server, :elixir_config,
 :elixir_counter, :elixir_sup, :erl_prim_loader, :error_logger, :file_server_2,
 :ftp_sup, :global_group, :global_name_server, :hackney_manager, :hackney_sup,
 :hex_fetcher, :httpc_handler_sup, :httpc_hex, :httpc_manager,
 :httpc_profile_sup, :httpc_sup, :httpd_sup, :inet_db, :inets_sup, :init,
 :kernel_safe_sup, :kernel_sup, :rex, :ssl_listen_tracker_sup, :ssl_manager,
 :ssl_sup, :standard_error, :standard_error_sup, :tftp_sup, :tls_connection_sup,
 :user]
NoRedInk makes learning grammar fun and easy

Wow, there’s kind of a lot going on in there! Most of what we see are things that Elixir setup to get things ready for our code to run.

You may notice that the list doesn’t include any mention of our HTTPoison dependency. That’s because HTTPoison is a thin Elixirifying wrapper over an Erlang HTTP client library called hackney. If you reexamine the list above you will find a couple of hackney processes, including :hackney_manager.

We’ve reached the key idea of how Elixir projects work.

Inside our Task, we called a function to fetch the page content: HTTPoison.get!(). That executed in our process. But we now have strong reason to believe that a separate hackney process is what actually did the fetching over the wires. If that’s true, these processes must have sent some messages to each other under the hood of these seemingly simple function calls.

Let’s make another debugging change to confirm our hunch:

defmodule TitleFetcher do
  # ...

  defp fetch_title do
    :sys.trace(:hackney_manager, true)

    body = HTTPoison.get!("https://www.noredink.com/") |> Map.get(:body)
    Regex.run(~r{<title>([^}, body, capture: :all_but_first)
    |> hd
    |> IO.puts

    System.halt
  end
end

This time the line added turns on a debugging feature of the OTP (the framework our code is running on top of). We’ve asked for a message trace() of that :hackney_manager process that we found earlier. Observe what that reveals:

$ mix run --no-halt
Compiled lib/title_fetcher.ex
*DBG* hackney_manager got call {new_request,<0.152.0>,#Ref<0.0.5.1103>,
                                   {client,undefined,hackney_dummy_metrics,
                                       hackney_ssl_transport,
                                       "www.noredink.com",443,
                                       <<"www.noredink.com">>,[],nil,nil,nil,
                                       true,hackney_pool,5000,false,5,false,5,
                                       nil,nil,nil,undefined,start,nil,normal,
                                       false,false,false,undefined,false,nil,
                                       waiting,nil,4096,<<>>,[],undefined,nil,
                                       nil,nil,nil,undefined,nil}} from     <0.152.0>
*DBG* hackney_manager sent {ok,{1457,541907,594514}} to <0.152.0>, new state     {...}
NoRedInk makes learning grammar fun and easy
*DBG* hackney_manager got cast {cancel_request,#Ref<0.0.5.1103>}

Bingo. The :hackney_manager process did indeed receive a message asking it to grab the web page we wanted to load. You don’t see it here, but hackney eventually sent a message back to the requester containing the page content.

This is how Elixir works. Concerns are divided up into processes that communicate by sending messages to each other. These processes are the “objects” of this world. They are what we should be judging to evaluate Elixir’s object-oriented merits.

Get to the Score Already!

We can assess how Elixir stacks up as an object-oriented language point by point using all of the definitions mentioned earlier. First up is the idea of “biological cells… only able to communicate with messages.” I hope the example above has shown that this is how you get things done in Elixir. In several other languages, objects are the recommended design tool, but in Elixir processes are essential.

What about the idea of “local retention and protection and hiding of state-process,” which we often call encapsulation? To put it bluntly, it’s enforced. Processes share nothing. Outside of the debugging tools, this is just the law of the land in Elixir.

Inheritance? No. Elixir doesn’t have this concept. It’s possible to build your own version on top of processes, but nothing is provided. This might be another plus.

There’s one last point: “extreme late-binding of all things.” We have tried to shorten this idea to polymorphism, but that’s not actually the same thing. Let’s sort this out with another example.

We’re just going to play around in Elixir’s REPL (iex) this time. Let’s ask Elixir to print something to the screen:

iex(1)> IO.puts("Printing to the terminal.")
Printing to the terminal.
:ok

That looks pretty straightforward, but we want to know how it works.

IO.puts() takes another argument that we got a default for above. Let’s add that in:

iex(2)> IO.puts(Process.group_leader, "More explicit.")
More explicit.
:ok

This version says that we want to send the output to the group_leader() of our process. That’s the same thing the default did for us in the earlier example. In the case of our REPL session, the group_leader() is essentially pointing at STDOUT. But what does group_leader() really return, in terms of Elixir data types?

iex(3)> pid = Process.group_leader
#PID<0.26.0>
iex(4)> IO.puts(pid, "It's processes all the way down!")
It's processes all the way down!
:ok

It you’ve been paying attention, I suspect you’re not at all surprised to see that this is another process. Why is that the case though, really?

The phrase “extreme late-binding of all things” is a way of saying something like, “I want the computer to sort out what code to actually run as the need arises.” In some language that might mean two different kind of objects could have print() methods and the computer decides which one to invoke based on which object the message is sent to.

In Elixir, this kind of late dispatch is simply sending messages to some kind of process that understands them. If we swap out that process with another one that understands the same messages, we can change what code is run. For example, we could capture the output from IO.puts():

iex(5)> {:ok, fake_terminal} = StringIO.open("")
{:ok, #PID<0.71.0>}
iex(6)> IO.puts(fake_terminal, "Where does this go?")
:ok
iex(7)> {inputs, outputs} = StringIO.contents(fake_terminal)
{"", "Where does this go?\n"}
iex(8)> outputs
"Where does this go?\n"

StringIO.open() creates a new process that pretends to be an I/O device. It understands the messages they receive and reply with, but it can substitute input from a string and capture output written into a separate string. This is possible because Elixir doesn’t care how the process works internally. It just delivers the messages to it. What code to run is sorted out inside the process itself as it receives the messages.

For a language we don’t traditionally think of as object-oriented, Elixir scores surprisingly well against a few different definitions. Processes make darn good objects!

Does This Matter?

Is it actually significant which language we decide to crown as “The Most Object-Oriented?” Of course not. It can be interesting to consider the parallels though.

If you shift your focus just right when looking at Elixir code, you can see the objects pop out. They’re a little different in this world. However, and this is the neat bit if you ask me, your object-oriented design skills (assuming you studied the good stuff, like Practical Object-Oriented Design in Ruby) still very much apply. I didn’t expect that and it’s a nice surprise.


James Edward Gray II
@JEG2
Engineer at NoRedInk

On-boarding as a New Remote Engineer Think about your on-boarding process. I don’t mean the...

$
0
0

On-boarding as a New Remote Engineer

Think about your on-boarding process. I don’t mean the part where HR or an office manager gives out paperwork. I mean the part where the new hire does actual engineering work. Do you have a fellow engineer pairing with the new hire through the process? Do you have a checklist to let the new hire know the scope of on-boarding and when they are finished? Do you ask the new hire to join in on the conversation to continuously improve the on-boarding process afterward? Do you set expectations for the first few weeks? If you didn’t answer positively to all of those questions, your on-boarding process can be improved.

After working for a few different startups over the years, and working in two more completely different industries, I have been through a wide variety of on-boarding processes. Some were intricate. Some were non-existent. But all were about me getting up to speed on my own. I could ask for help with things here and there at each place. However, the general vibe was that you needed to be up to speed as fast as possible for the sake of the company.

The First Day

Things are different at NoRedInk. My on-boarding experience has been less focused on what I personally need to do to get up to speed. It has been more focused on what we do as a whole and how I contribute to this process.

Rather than telling me what to do and seeing if I can do it, the first day was all pairing with another engineer. We ran through the administrative work of setting up accounts for services and credentials together. Doing the administrative work together was great because I had the chance to ask questions about what the different services did. More importantly, we had a chance to discuss why we chose each service and how we generally use them in day to day engineering.

We did actual engineering work next. My pair walked me through the steps of choosing an available issue, style conventions, and how to finish up on github. All of this information is overwhelming to take in on the first day if you are by yourself. To help alleviate some of this cognitive load, we have to talk about one of the most amazing things about being at NoRedInk: wikis!

Wikis!

We have just about everything documented somewhere in a wiki. Imagine your employee handbook but better. There are five main wikis: Engineering, BizOps, Sales, Product, and Non-Technical. If you are curious about how sales does their demos, you can find an entry that details the whole process. If you want to know how content gets created on the site, there is an entry in the Non-Technical wiki. Practically anything you can think of is in a wiki somewhere! But wikis are worthless if the information becomes stagnant. So we constantly update entries and make changes to keep the information current.

Let’s focus on the engineering wiki. The first thing to know is that even though it is an engineering wiki, the information is mostly about process rather than algorithms. Some examples are: “The Best Practices for Discussions”, “The Communication Policy for Email, Github, Slack”, and “The Engineer Onboarding Checklist”. Yep, we keep the checklist of things to do during on-boarding in our wiki! To keep the checklist current, the last item is to update the checklist with any information that might have changed.

Our technical stuff is more like what you would expect in an engineering wiki: best practices for front-end, style guides, domain information. With the information in the wiki and out of the heads of individuals, it is much easier to ensure everyone understands what the process currently is. We also can ensure we are following an up to date process. If we decide to make a slight change to our pull request process, it can happen in the wiki and everyone can be aware of it.

The biggest benefit I get from having so much information written down is being able to understand it at my own pace. There are some familiar bits that I get the first time around. Then there are some unfamiliar bits that I need to read multiple times to really understand. In day to day work, it means that I don’t have to bother someone else when I have a simple question. I can find the information in the wiki and answer my own question. As a remote worker, it is a great feeling to know that you can get most of the information you need at any time.

Remote Culture

The pairing doesn’t stop with on-boarding. Every day that I have been at work, I have paired with someone on something engineering related. About 80% of the time it is technical work, but sometimes we do pair on non-technical work. Pairing can be a quick 15 minute session to work through a bug, or it can be a full 8 hours if you (and your pair) want. I have paired with people in the office and my fellow remote workers.

One of the big issues with working remote is losing the connection with your co-workers. NoRedInk is the most remote friendly place I have worked. Co-located people will often ask remote people if they would like to pair, and vice versa. If a discussion happens in-office that concerns multiple people, it will usually move to email, github, or slack. The purpose is to keep everybody in the loop, whether they are working in-office or remotely.

Much of our engineering process is centered around asynchronous communication. This is a huge win for working remotely! The slack channels are always abuzz with information and ideas. But there are times when a face-to-face call makes more sense than typing. In these cases, we will throw up a quick five minute hangout, fire-up screenhero or even use a regular old telephone. Once we hash out the details, we report back in the channel on what the conclusion is. There’s no sense in forcing asynchronous communication if a synchronous call is more effective. Again, the focus is on communicating effectively and keeping everyone connected.

To make sure the different teams stay connected, we have a weekly team lunch. The in-office folk gather in a hangout with the remote folk and someone presents on a topic. There is a projector in office (as most offices might have) so people don’t have to crowd on a single screen. We have presented on what the engineering team is doing, how the process is changing within the product team, what is going on with sales, and less company related stuff like OSX productivity tips.

We have expanded so much as a team lately that during our weekly lunches the external in-office microphone couldn’t keep up. We had so many people around the lunch table that it was hard for the remote people to hear the conversation. To make audio better for us joining remotely, a second external microphone was bought. We can hear much better now when someone talks at the far end of the table. Little things like adding an additional microphone go a long way to making remote people feel like they are part of the office.

One of Us

There is quite a bit more that is great about working at NoRedInk. Rather than reading more in this blog post, you should come experience it for yourself. We are always looking for great people to join and help us make the team even better. Check out the jobs page to see if something fits. Or if you have tips on where we can improve our process, I’d love to hear about it. Send me a tweet at @st58 with your suggestions.

Hardy Jones
@st58
Engineer at NoRedInk


Our Engineering Hiring Process

$
0
0

Hiring is something that we care deeply about at NoRedInk. We aim to have a process that’s reliable and efficient, and that provides a positive and respectful experience for the candidate

If you’re thinking about applying, this should help you learn what to expect from each step of the process.

The Interview Steps

  • Application
  • Take Home Challenge
  • Conversation with Engineer
  • Technical Interviews
  • Conversation with Director or CEO
  • Lunch with Team

Application

You can find details about the engineering positions we’re hiring for on our jobs page.

For each position, the expertise and qualities we’re looking for are described in its job post. At this initial stage, we’re looking to see whether your resume and cover letter match the requirements of the job you’re applying for.

This is also a great opportunity for you to tell us a little about yourself and why you want to join NoRedInk. Although a cover letter isn’t required, it’s definitely a nice way for us to learn a bit more about you.

Take Home Challenge

Our take home challenge has a two-hour time limit and can be taken on your schedule.

We’ve built a simple web app to administer all take home submissions. This allows you to log in and start the challenge at any time. When you submit it, your solution is sent to one of our engineers to be evaluated. These evaluations are “blind” in that the evaluator doesn’t have access to any personally identifiable information about who they’re evaluating, such as name or resume.

There are two reasons for the two-hour window over an unlimited window for working on the challenge:

  • It helps us standardize solutions across all candidates applying for the same position, which helps us make fair evaluations.
  • It helps us be respectful of the candidate’s time.

We recommend using the language and, if necessary, framework you’re most familiar with. We can almost surely find an engineer on our team that’s skilled enough at it to evaluate your solution fairly.

Conversation with an Engineer

The conversation is scheduled for thirty minutes. We’ll ask you non-technical questions so that we can learn more about you as a candidate and how you’d fit as an employee.

You’ll also have time to ask us questions about your interviewer, the company, or anything else you want to learn about us. If you can, prepare your questions beforehand.

Technical Interviews

We do three technical interviews. Each one lasts for one hour and thirty minutes, plus an additional thirty minutes that you can use to ask questions.

The first technical interview is done remotely. The two following interviews are contingent on the first and happen on a different day. These are on-site for candidates applying to work from our SF offices and remote otherwise.

We never ask you to write code on a whiteboard, because we’re not hiring you to write code on a whiteboard. We ask you to bring your own laptop, with your development environment of choice, to use for 100% of the technical interview process. We want to understand what it’d be like to work with you, so the closer your environment is to what you’d normally use on the job, the better.

We also encourage you to use the same tools in the interview as you would on the job. Please Google whatever you like; it’d be weird to pretend like you didn’t have an Internet connection when you clearly would on the job. Restricting your normal toolbox would give us a less accurate picture of what it’d be like to work with you.

Our goal is to test skills you use on a daily basis, as opposed to computer science trivia you wouldn’t actually use on the job. Still, we recognize that you may not be used to solving problems from start to finish in such a limited timeframe. A good way to get comfortable with time limits is to do some problems on Exercism.io with a self-imposed time limit of an hour. James Gray gave a talk at Railsconf 2016 with other tips on interview preparation.

Content-wise we’ll test the technologies and skills you’ll use in your day-to-day work. For example, if you’re applying for a Front-End job, we’ll want you to demonstrate experience with JavaScript, HTML and CSS.

Conversation with Director or CEO

This is the final interview and it’s also non-technical. It lasts for two hours with one hour and thirty minutes of interview and thirty minutes for your questions. We’ll want to learn about details from your work experience and what makes you a great engineer.

This is also an excellent time to ask questions about the company’s strategic plans, vision, and values, but you are free to ask any questions you like.

Lunch with Team

On the same day as your final interview, you’ll have lunch with the team so you can get to know your potential coworkers. This is not an interview, but just an opportunity for you to get to know us better. There’s no evaluation for this step on our end.

Timeline

We pride ourselves on moving extremely quickly with our interviewing process, but a lot of the speed depends on your availability. Interviewing takes priority over all other work here at NoRedInk, and we can usually accommodate interviews at a day’s notice.

In practice, it takes two to four weeks from your application until the offer. We evaluate most resumes and take home challenges within 24 hours and we’ll be available for your first technical interview within 48 hours.

Conclusion

Creating a good interview process is hard, especially as a startup.

Increasing the transparency of our hiring process is important not only so that you know what to expect, but also so we can make our process better.

We hope learning about our process makes you feel more prepared and confident. If you have any questions about our process or the company, please feel free to get in touch.

Ready to apply? Apply now on our jobs page!


Marcos Toledo
@mtoledo
Director of Engineering at NoRedInk

Writing Friendly Elm Code

$
0
0

So you’ve chosen to use an elegant language designed with the programmer in mind. You’ve followed the Elm Architecture tutorial, so you understand how to structure an application. You’ve written some Elm code. Maybe you’ve even read other people’s Elm code, or taken over someone else’s Elm app. Whatever your level of experience in Elm, and despite the work the language puts in to keep your code readable and tidy, it’s possible to write some deeply unfriendly Elm code.

So why write “friendly” code? For us, the first people we have in mind when we are writing code are the students and the teachers who use NoRedInk. But we’re also writing code for each other–for other engineers–rather than writing code for just ourselves. Writing readable code can be hard, especially since not everyone agrees what is or is not readable. On my team, there are different preferences as to whether doc comments, type signatures, or descriptive naming are most important when encountering a new chunk of code. I don’t want to make an argument there (#descriptive_naming), but over the course of working with and writing Elm components of various sizes and complexities, I’ve found some general guidelines to be helpful.

I recommend that Elm programmers don’t be fancy, limit case/if..then expressions, and think of the Elm Architecture as a pattern. Obviously my opinions are fact, but feel free to have your own in public at me @t_kelly9.

Being Fancy

Being Fancy is fun. Doing something tricky, or unexpected, can feel like you’ve found The Neatest Way to solve a problem. But being fancy when modeling data makes it harder to understand that data. For a functional language, in which it’s likely you’re doing a lot of mapping and filtering, creating complex types is likely to cause frustration. Writing or modifying decoders for particularly complicated types is likely to cause active sorrow and regret.

Here’s an enthusiastically contrived example of over-complicated types:

Suppose we have a commenting component that we made for use on our media website. Users can comment on books, songs, pictures, and videos. So far, we’ve only wanted users to be able to comment with plaintext, leading to a hijacking of our platform as a means of showcasing ascii art. Our model looks like this:



type alias Model =
    { mediaUrl : String
    , description : String
    , currentUser : User
    , comments : List Comment
    }

-- Comments --

type alias Comment =
    { commenter : User
    , comment : String
    , created : Date
    }

-- User --

type alias User =
    { name : String }

Embracing our userbase’s love of self-expression through images, we’ve decided to try out allowing users to comment only in the form of the media on a given page. That is, for books, the commenting system remains unchanged–users comment in words. For songs, users can comment by uploading an audio file. And so on.

Perhaps sensibly, perhaps not, we decide to create a type that describes the media type on a given view, so that we don’t get confused later and accidentally allow an audio comment on a video (that would be crazy!).



type alias Model =
    { mediaUrl : String
    , description : String
    , currentUser : User
    , comments : List MediaComment
    }


-- Comment --

type alias Comment =
    { commenter : User
    , comment : String
    , created : Date
    }


type MediaComment
    = Book Comment
    | Song Comment
    | Picture Comment
    | Video Comment


-- User --

type alias User =
    { name : String }

But at this point, we still only actually support comments of type String. Is that what we want? Maybe–we can use the comment field to store an actual comment’s text content for book comments, and then just a source url otherwise, but that’s pretty simplistic. What if we want our video comments to have poster images that are attached to video comment?

Really, what we want to do is extend the idea of a Comment record with a meta idea about a comment and the comment’s contents. Sounds like record extensibility, right?



type alias Model =
    { mediaUrl : String
    , description : String
    , currentUser : User
    , comments : List MediaComment
    }


-- Comment --

type alias Comment a =
    { a
    | commenter : User
    , created : Date
    }


type alias StringCommentContents =
    { comment : String }


type alias UrlCommentContents a =
    { a
    | src : String
    }


type alias TypeCommentContents =
    { type' : String }


type alias PosterCommentContents =
    { posterSrc : String }


type MediaComment
    = Book (Comment StringCommentContents)
    | Song (Comment (UrlCommentContents TypeCommentContents))
    | Picture (Comment (UrlCommentContents {}))
    | Video (Comment (UrlCommentContents PosterCommentContents))


-- User --

type alias User =
    { name : String }

Using record type extensibility may seem like a great idea, since the goal is to “extend” a type. But if your brain is remotely like mine, reading the above code was a frustrating experience. We’re not modeling a very complex system, but in the interest of keeping code DRY we’ve very quickly ended up in a brain-stack-overflow situation.

So how can we get ourselves out of this mess, and back to a friendly description of our commenting system? We “make a hole” rather than say “this thing is like this thing which is like this thing in these ways.”

Here, we maintain the idea of the “Media Comment,” which protects against accidentally using the wrong comment view for a given media type, but we use the “make a hole” strategy.



type alias Model =
    { mediaUrl : String
    , description : String
    , currentUser : User
    , comments : List MediaComment
    }


-- Comment --

type alias Comment a =
    { commenter : User
    , created : Date
    , comment : a
    }


type alias StringCommentContents =
    { comment : String }


type alias UrlCommentContents a b =
    { src : String
    , type' : a
    , posterSrc : b
    }


type MediaComment
    = Book (Comment String)
    | Song (Comment (UrlCommentContents String ()))
    | Picture (Comment (UrlCommentContents () ()))
    | Video (Comment (UrlCommentContents String String))

We can, of course, flatten this all the way back out if we want, but continue to use the “make a hole” strategy:



type alias Model =
    { mediaUrl : String
    , description : String
    , currentUser : User
    , comments : List MediaComment
    }


-- Comment --

type alias Comment a b c d =
    { commenter : User
    , created : Date
    , comment : a
    , src : b
    , type' : c
    , posterSrc : d
    }


type MediaComment
    = Book (Comment String () () ())
    | Song (Comment () String String ())
    | Picture (Comment () String () ())
    | Video (Comment () String String String)


-- User --

type alias User =
    { name : String }

This is better than the version using tons of extensibility, but there’s still too much complexity to comfortably keep track of. We can try decoupling our what-kind-of-media-idea from our what-a-comment-looks-like idea:



type alias Model =
    { mediaUrl : String
    , mediaType : Media
    , description : String
    , currentUser : User
    , comments : List Comment
    }


-- Comment --

type alias Comment =
    { commenter : User
    , created : Date
    , comment : Maybe String
    , src : Maybe String
    , type' : Maybe String
    , posterSrc : Maybe String
    }


type Media
    = Book
    | Song
    | Picture
    | Video


-- User --

type alias User =
    { name : String }

Here it becomes more obvious that we’ve been neglecting the information in our model about our actual media type, but leaving that aside, there are a couple of things to notice here. One, it’s the most succinct. Two, information about the necessary and expected shape of a given comment is lost–view code written with this code is going to be full of Maybe.map text model.comment |> Maybe.withDefault (text "oh no we're missing a comment how did this happen???")s. Three, it’s easy to understand what fields exist, but hard to know which fields are expected/mandatory/in use.

A final option for organizing this code: don’t try so hard to be DRY. Have different models/views for working with different comment types, and don’t worry about having overlap in those field names when your record is describing different shapes.



type alias Model =
    { mediaUrl : String
    , description : String
    , currentUser : User
    , comments : List MediaComment
    }


-- Comment --

type alias Comment a =
    { commenter : User
    , created : Date
    , content : a
    }


type MediaComment
    = Book (Comment String)
    | Song (Comment { src : String, songType : String })
    | Picture (Comment { src : String })
    | Video (Comment { src : String, videoType : String, posterSrc : String })


-- User --

type alias User =
    { name : String }

Whatever you decide, do be aware of the outsize impact in complexity/brain-case-space when using extensible records over some other options. Extensibility is for writing flexible functions, not for modeling your types. e.g., for our final example above, we could add another type that would make writing type signatures simpler, without making it harder for us to think about our model:



type alias UrlComment a =
    { a | src : String }

Again, this is a pretty contrived example (for as simple a concept as a comment, it probably makes the most sense to just separate out comment meta data from comment contents). For a complicated web application, though, it’s not unlikely to run into very complex structures on the frontend that can’t be easily broken down into the categories of “shared-shape-meta-data” and “differing-content-data.” My hope here is just that if you find yourself in a situation where your data modeling is getting out of hand across different pages (i.e., tons of different ways of representing very similar but non-identical information), you’ll be able to simplify your models without too much confusion.

case/if..then Expressions

Case expressions are awesome. Elm pattern matching is basically insane in a good way. Using case expressions cleverly can mean cutting down on extraneous/hard-to-follow if/else branches, but using too many case expressions can become hard to deal with.

Let’s make a view with a couple of steps to it. Say we’re making a game. We’re going to have a welcome screen, a game-play screen, and a game-over screen. Refreshing the page is clear cheatery, so let’s not worry about persistence. We’re also not going to worry about game logic. All we care about are the flow steps.

This is our model:



type alias Model =
    { playerName : String
    , moves : List ( Int, Int )
    , boardSize : (Int, Int)
    , gameStep : GameStep
    }

type GameStep
    = Welcome
    | GamePlay Turn
    | GameOver


type Turn
    = Player
    | ComputerPlayer

Let’s start out with a naïve view (naïve doesn’t mean don’t do this! It just means don’t stop your work here). Skipping game logic means there’s not much use to this view, but that should help us to focus in on good patterns to follow and less-good patterns to minimize.



view : Model -> Html Msg
view model =
    div
        []
        [ viewHeader model
        , viewGameBoard model
        ]


viewHeader : Model -> Html Msg
viewHeader model =
    header [] (headerContent model)


headerContent : Model -> List (Html Msg)
headerContent {gameStep, playerName} =
    case gameStep of
        Welcome ->
            [ div [] [ text ("Welcome, " ++ playerName ++ "!") ] ]

        GamePlay turn ->
            case turn of
                Player ->
                    [ div [] [ text ("It's your turn, " ++ playerName ++ "!") ] ]

                ComputerPlayer ->
                    [ div [] [ text "It's the computer's turn. Chillll." ] ]

        GameOver ->
            [ div [] [ text "Game Over!!!" ] ]


viewGameBoard : Model -> Html Msg
viewGameBoard model =
    case model.gameStep of
        Welcome ->
            text ""

        GamePlay turn ->
            buildBoard model.boardSize turn

        GameOver ->
            div [] [ text "This game ended. We're skipping game logic so who knows who won!" ]


buildBoard : (Int, Int) -> Turn -> Html Msg
buildBoard boardSize turn =
    let
        squareStyles =
            case turn of
                Player ->
                    style [ ("border", "1px solid green") ]

                ComputerPlayer ->
                    style [ ("border", "1px solid red") ]

    in
        tbody [] (buildBoardRow boardSize squareStyles)


buildBoardRow : (Int, Int) -> Attribute Msg -> List (Html Msg)
buildBoardRow (boardWidth, boardHeight) squareStyles =
    viewBoardSquare squareStyles
        |> List.repeat boardWidth
        |> List.repeat boardHeight
        |> List.map (tr [])


viewBoardSquare : Attribute Msg -> Html Msg
viewBoardSquare squareAttribute =
    td [ squareAttribute ] [ text "[ ]" ]

Okay, cool! So that works, as long as we’re fine with having an un-updateable model with corresponding view.

But there are a couple of things that are bad. One, we’re repeating case expressions based on game step all over the place. Game state is a very top level concern. We shouldn’t have to re-evaluate what step we’re on all over the place. Another less-than-stellar thing we’re doing is nesting case expressions. It’s harder to follow and, as a rule, not that necessary.

See if you like this view better:



view : Model -> Html Msg
view model =
    div [] (buildBody model)


buildBody : Model -> List (Html Msg)
buildBody {gameStep, playerName, boardSize} =
    case gameStep of
        Welcome ->
            [ viewHeader ("Welcome, " ++ playerName ++ "!") ]

        GamePlay Player ->
            [ viewHeader ("It's your turn, " ++ playerName ++ "!")
            , buildBoard boardSize "green"
            ]

        GamePlay ComputerPlayer ->
            [ viewHeader "It's the computer's turn. Chillll."
            , buildBoard boardSize "red"
            ]

        GameOver ->
            [ viewHeader "Game Over!!!"
            , div [] [ text "This game ended. We're skipping game logic so who knows who won!" ]
            ]


viewHeader : String -> Html Msg
viewHeader headerText =
    header [] [ text headerText ]


buildBoard : (Int, Int) -> String -> Html Msg
buildBoard boardSize boardColor =
    tbody [] (buildBoardRow boardSize boardColor)


buildBoardRow : (Int, Int) -> String -> List (Html Msg)
buildBoardRow (boardWidth, boardHeight) boardColor =
    viewBoardSquare boardColor
        |> List.repeat boardWidth
        |> List.repeat boardHeight
        |> List.map (tr [])


viewBoardSquare : String -> Html Msg
viewBoardSquare boardColor =
    td
        [ style [ ("border", "1px solid " ++ boardColor) ] ]
        [ text "[ ]" ]


It’s more succinct, but more importantly, it has wayyy less branching logic. All the logic over what to show/what not to show is at the top level, so it’s very easy to see what’s going on and what is going to be rendered to the page. This pattern allows for more easily generalizable components. To recap, the two moves we made here were to branch based on state at the top of our view and to eliminate sub-branching logic by making use of Elm’s pattern matching for tuples.

Hopefully, using this pattern will make it easier to extract pieces of your view for reuse, since they’ll end up being dependent on much less of the model than they would otherwise. Compare the board building method of our first try to our second try–the second one just wants to know how big and what color, and it’ll make you a board, while the first would like to know whose turn it is, please and thanks.

Elm Architecture as a Pattern

My final tip: don’t forget that the Elm Architecture is just a bunch of functions that sometimes get called! That is, update is just a function, and calling it recursively is totally fine if you want to.

Maybe the Cookies API should not exist

$
0
0

One of the unique things about working at NoRedInk is that every other week we have “Elm Friday” where I pair with an engineer on something. Tessa and I worked on an alternate Array implementation. Noah and I made the navigation package for single-page apps. Scott helped create the react-elm-components package for embedding Elm in React apps. I think “Elm Friday” is wildly productive because each person brings unique perspective, interests, skills, knowledge, etc. and we augment each other. All of these are projects that are (1) great for the overall ecosystem and (2) projects that we would probably not have worked on individually.

Point is, Richard and I recently found ourselves working on elm-lang/cookie and came to a surprising conclusion: maybe it should not exist!

What are Cookies?

A cookie is a little piece of information associated with your domain. The important detail is that all that info gets collected and put into the Cookie header of every request you make. So if someone sets theme and user cookies, you would have a header like this on every HTTP request:

GET /spec.html HTTP/1.1 Host: www.example.org Cookie: theme=light; user=abc123 …

How do you create cookies though? One way is to use the Set-Cookie header when your server sends the page in the first place. For example, the server could set the theme and user cookies by providing a response like this one:

HTTP/1.0 200 OK Content-type: text/html Set-Cookie: theme=light Set-Cookie: user=abc123; Expires=Wed, 09 Jun 2021 10:18:14 GMT …

Okay, so that is the basic functionality. On top of that, JavaScript provides a truly awful cookie API that can be understood as exposing the Set-Cookie header functionality at runtime.

We are in the process of exposing all these Web Platform APIs so you can use them from Elm directly. Just doing a thin wrapper around JS APIs is easy and quick, but the whole point of this language is trying to do better. So I want to provide the functionality in a way that is delightful and works well with Elm. I think the websocket library is a great example of this! So in approaching cookies, we asked ourselves:

  • What exactly are the capabilities provided by the browser?
  • How do we expect people to use these capabilities?
  • Are there traps that people are likely to fall into?
  • How can we design our API such that “doing the right thing” is the default?

In the end, a great platform API fits into The Elm Architecture and the overall ecosystem in just the right way. Ideally users do not even realize that there were other options. They just are writing oddly solid apps and feeling pretty good about it.

As we explored these questions, it revealed that almost everything about cookies is a trap.

Cookies have Problems

Security

You may have heard of “tracking cookies” that let companies see which pages you have visited. Here is how that works:

  1. A company convinces as many websites as possible to embed an “Like button” by adding a simple <script> tag to their page. Share this recipe on Facebook, Twitter, and Pinterest!

  2. The <script> can load arbitrary code that runs with full permissions. The have access to everything from the document to whatever parts of your JS code are globally accessible.

The ultimate goal of this setup is to uniquely identify a user, which is where cookies come in.

Say you visit hooli.com. They set a cookie that uniquely identifies you. Later you go to a blog with a “Hooli Share” button which is embedded as a <script>. They can run some code that figures out what URL you are visiting, how long you are there, etc. When they have all the info they want, they send an HTTP request to hooli.com which automatically gets any hooli.com cookies. That way they get the data tagged with your unique identifier. That means they know exactly who you are as well as what sites you are visiting. Pretty neat trick!

Now, I had a vague notion that people track me online, but before looking into cookies, I had no idea it was so easy to get so much information. So this seems like a pretty bad problem to me, but I suspect enough money is made off this that it is likely to continue to exist.

Memory Caps and Encoding

When I first described cookies up above, I used a theme cookie. The idea there was that we have some app with a light and dark theme. Maybe people want a dark theme at night or something! But does it make sense to store that information in a cookie? Probably not.

First problem, browsers cap the memory available for cookies. Some person on the internet suggests that the limit is 4093 bytes for your whole domain. Pretty tiny! It sounds like in the olden times, when you got to the max size, you just had to wait until some cookies expired. Now it sounds like it will silently evict old cookies. Either way, pretty bad.

Second problem, cookies can contain a very limited set of characters. For example, Safari permits ASCII characters only. In the version we tested, it just ignores your attempt to set cookies if it sees any characters it does not like. So if you have a string with Unicode characters outside that range, it will break in various ways depending on the browser.

So for a vast majority of “storage” scenarios, this API is significantly weaker than localStorage and sessionStorage.

Note: Using a cookie would also mean that the data is attached to every single HTTP request you make. As we learned, that is not too much data, but why bother sending it if the whole point of browser storage is that you do not handle it in your server?

A Better Way?

As far as I can tell, the only usage that really makes sense is the following scenario:

A user comes to your website http://mail.example.com and you want to serve them their inbox immediately, no sign in necessary.

In that case, you do want to have some additional information added by the browser that can help reliably identify and authenticate the user. You want to know that and nothing else.

Based on a unique identifier, you can look up things like theme from your database and send back the appropriately styled HTML. Basically, any other information can be derived from a unique ID. In this world, people would have a consistent experience no matter what device they log in from. You also own the customization data, so if you do an upgrade such that theme works different now, you can change it all at once on your servers. You do not have to wait for the user to sign back in, keeping crazy fallback code forever.

So it sounds like the following constraints could help:

  • A domain can only store a unique identifier.
  • The only way to set this unique identifier is with an HTTP response header, like a restricted Set-Cookie.
  • That unique identifier is only added to HTTP requests headers if the page was served from the same domain the request is going to.

That means I can log in on hooli.com without Hooli permanently knowing who I am when I check out a French Toast recipe.

There are problems though. I use Google Analytics to learn how many people are checking out Elm, and it is actually pretty handy that they can distinguish between visits and unique visitors. One person visiting 10 times is very different than 10 people visiting one time each! I think I could still get that information, but my servers would have to have some extra logic enabled to assign unique IDs. So it would be a bit harder to set up, but for inocent questions like, “how can the Elm website be more helpful?” it seems like this scheme would still work.

It seems like there is a lot of money in keeping the current design, so who knows if something like this will ever happen!

What to do in Elm?

It was getting pretty late on Friday by the time Richard and I really loaded all this information into our brains. We had drafted a pretty nice API and were kind of tired based on how insane everything was.

As we were finalizing the API and writing documentation, I asked Richard, if you only want a unique identifier, and you only want to set it with a header like Set-Cookie, why are we even doing this? The one valid scenario did not require this API! Neither of us could think of a compelling reason to set cookies any other way. Especially considering that the browser’s localStorage API covers the only other plausible use with a higher data cap and proper unicode support.

Elm is very serious about having the right defaults. People should just do the easy and obvious thing, and somehow, as if by accident, end up with a codebase that is quite nice. So libraries need to be pleasant to use, of course, but they also need to rule out bad outcomes entirely. And as far as we can tell, this means the cookie library should not exist!

Note: Richard and I could not think of legitimate uses for this API, but that may be a lack of creativity or experience. Open a friendly issue here if you think you have a scenario that cannot be covered by the Set-Cookie header.


Evan Czaplicki
@czaplic
Engineer at NoRedInk

Functional Randomization

$
0
0

When I first started playing with Elm, the discoveries went something like this:

  1. Wow, this is some powerful functional mojo.
  2. I bet this would be a blast to make games with!
  3. I just need a random number…
  4. I better go read half of the Internet…
  5. Have I made bad life choices???

Laugh all you want, but back then Elm’s documentation of the Random module opened with a twenty line example. I was using Ruby everyday where I had rand()!

Why are functional programming and random number generation mortal enemies? The answer is kind of interesting…

Seeds

To make sense of why this topic is complicated, let’s begin with some exploration in a seemingly less strict language. Here’s the easiest way to start pulling random numbers out of Elixir (or Erlang):

$ iex
Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.2.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :rand.uniform
0.006229498385012341
iex(2)> :rand.uniform
0.8873819908035944
iex(3)> :rand.uniform
0.23952446122301357

To prove those are really random, I’ll repeat the process and hope we see new numbers:

$ iex
Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.2.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :rand.uniform
0.9471495706406268
iex(2)> :rand.uniform
0.9593850716627006
iex(3)> :rand.uniform
0.19095631066267954

OK, they look different. I better pull a rabbit out of my hat quick, because this is shaping up to be one lame blog post.

Let’s ask for a random number one more time, but let’s keep an eye on the process dictionary as we do it this time. What’s a process dictionary you ask? It’s Elixir’s dirty little secret. Inside every process, there’s some mutable data storage. Oh yeah, believe it:

$ iex
Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.2.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Process.get_keys
[:iex_history]
iex(2)> :rand.uniform
0.6585831964907924
iex(3)> Process.get_keys
[:iex_history, :rand_seed]

Look a rabbit, err, a :rand_seed!

Before we asked for a random number, the process dictionary only contained some data from my REPL session. But after the call, there was random number related data in there.

Let’s dig a little deeper by watching the seed as well pull another number:

iex(4)> seed = Process.get(:rand_seed)
{%{max: 288230376151711743, next: #Function<8.51808955/1 in :rand.mk_alg/1>,
   type: :exsplus, uniform: #Function<9.51808955/1 in :rand.mk_alg/1>,
   uniform_n: #Function<10.51808955/2 in :rand.mk_alg/1>},
 [41999190440545321 | 147824492011192469]}
iex(5)> :rand.uniform
0.5316998602996328
iex(6)> Process.get(:rand_seed)
{%{max: 288230376151711743, next: #Function<8.51808955/1 in :rand.mk_alg/1>,
   type: :exsplus, uniform: #Function<9.51808955/1 in :rand.mk_alg/1>,
   uniform_n: #Function<10.51808955/2 in :rand.mk_alg/1>},
 [147824492011192469 | 5427558722783275]}

What did we see there? First, seeds are gross looking things. More importantly though, it’s changing as we pull numbers. You can tell by comparing the numbers in the improper list at the end of the seed.

What are these changing seeds?

You can kind of think of random number generation as a giant list of good numbers available for picking. (Yes, I’m simplifying here, but stick with me for a moment…) It’s like someone sat down at the dawn of computing, rolled a dice way too many times, and carved all the numbers into stone tablets for us to use later on. The problem is, if we always started at the beginning of the list, the sequence would always be the same and video games would be hella boring!

How do we make use of this master list? We seed it! You can think of that as your computer closing its eyes, pointing its finger at a random spot in the list, and saying, “I’ll pick this number next.” Except that your computer probably doesn’t have eyes or a finger. Anyway, that’s how you get random numbers out of a random list… randomly. (At least it’s close enough for our purposes. I know I lied some. I’m sorry. It’s for a good cause.)

Getting back to our example, it turns out there’s another function that we could call to get random numbers. We were using :rand.uniform/0, but I think it’s time to try :rand.uniform_s/1. What does the _s stand for? I have no idea. Let’s pretend it’s “seeded” solely because it helps my story here and you want me to succeed. :rand.uniform_s/1 expects to be passed a seed and I stuck the first one we examined in a seed variable. Right after we saw that seed last time, we got the number 0.5316998602996328 out of the generator. You can scroll up if you think I would make something like that up. Or you could scroll down:

iex(7)> :rand.uniform_s(seed)
{0.5316998602996328,
 {%{max: 288230376151711743, next: #Function<8.51808955/1 in :rand.mk_alg/1>,
    type: :exsplus, uniform: #Function<9.51808955/1 in :rand.mk_alg/1>,
    uniform_n: #Function<10.51808955/2 in :rand.mk_alg/1>},
  [147824492011192469 | 5427558722783275]}}

The return value has changed from what :rand.uniform/0 gives us, but the first item in the tuple is the random number that we saw before. The second item in the tuple is a new seed. You’ve seen it before too. It was in our process dictionary after we generated 0.5316998602996328 last time. I can’t believe you didn’t recognize it!

When using the _s functions, Elixir leaves it to us to track the seeds. Anytime we fetch a number, it gives us that and a new seed for the next time we need one. It’s kind of like generating two numbers each time, one to use and one to be the new finger. Well, you know what I mean!

In Elixir, asking for a seed is an impure function that returns a different answer with each call. (Oops, I lied again! More on that later…) Again, compare the numbers at the end of each seed to see the differences:

$ iex
Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.2.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Process.get_keys
[:iex_history]
iex(2)> :rand.seed_s(:exsplus)
{%{max: 288230376151711743, next: #Function<8.51808955/1 in :rand.mk_alg/1>,
   type: :exsplus, uniform: #Function<9.51808955/1 in :rand.mk_alg/1>,
   uniform_n: #Function<10.51808955/2 in :rand.mk_alg/1>},
 [88873856995298220 | 199352771829937909]}
iex(3)> :rand.seed_s(:exsplus)
{%{max: 288230376151711743, next: #Function<8.51808955/1 in :rand.mk_alg/1>,
   type: :exsplus, uniform: #Function<9.51808955/1 in :rand.mk_alg/1>,
   uniform_n: #Function<10.51808955/2 in :rand.mk_alg/1>},
 [12382884451496331 | 275984276003718626]}
iex(4)> :rand.seed_s(:exsplus)
{%{max: 288230376151711743, next: #Function<8.51808955/1 in :rand.mk_alg/1>,
   type: :exsplus, uniform: #Function<9.51808955/1 in :rand.mk_alg/1>,
   uniform_n: #Function<10.51808955/2 in :rand.mk_alg/1>},
 [252423044625965483 | 35827924244841321]}

The :exsplus argument specifies the algorithm to use, which is probably the source of those anonymous functions we keep seeing in the seeds. I’ve used the default that Elixir would pick itself, if we let it choose the seed.

Now, using seeds is a pure function call that always yields the same results:

iex(5)> seed = :rand.seed_s(:exsplus)
{%{max: 288230376151711743, next: #Function<8.51808955/1 in :rand.mk_alg/1>,
   type: :exsplus, uniform: #Function<9.51808955/1 in :rand.mk_alg/1>,
   uniform_n: #Function<10.51808955/2 in :rand.mk_alg/1>},
 [256126833480978113 | 32213264852888463]}
iex(6)> :rand.uniform_s(seed)
{0.2372892113261949,
 {%{max: 288230376151711743, next: #Function<8.51808955/1 in :rand.mk_alg/1>,
    type: :exsplus, uniform: #Function<9.51808955/1 in :rand.mk_alg/1>,
    uniform_n: #Function<10.51808955/2 in :rand.mk_alg/1>},
  [32213264852888463 | 36180693784403711]}}
iex(7)> :rand.uniform_s(seed)
{0.2372892113261949,
 {%{max: 288230376151711743, next: #Function<8.51808955/1 in :rand.mk_alg/1>,
    type: :exsplus, uniform: #Function<9.51808955/1 in :rand.mk_alg/1>,
    uniform_n: #Function<10.51808955/2 in :rand.mk_alg/1>},
  [32213264852888463 | 36180693784403711]}}
iex(8)> Process.get_keys
[:iex_history]

Also, notice that we’re no longer using secret mutable storage to generate numbers this way.

How do we use all of this in actual work? We don’t. We typically just cheat and call :rand.uniform/0. But now you know how the cheating works!

For the sake of example, and blog post line count, here’s a simple script doing things The Right Way™. It streams random numbers, a new one each second:

defmodule ARandomNumberASecond do
  defstruct [:callback, :rand_seed]

  def generate(callback) when is_function(callback) do
    state = %__MODULE__{callback: callback, rand_seed: :rand.seed_s(:exsplus)}
    do_generate(state)
  end

  defp do_generate(
    state = %__MODULE__{callback: callback, rand_seed: rand_seed}
  ) do
    {number, new_rand_seed} = :rand.uniform_s(rand_seed)
    callback.(number)

    :timer.sleep(1_000)

    new_state = %__MODULE__{state | rand_seed: new_rand_seed}
    do_generate(new_state)
  end
end

ARandomNumberASecond.generate(fn n -> IO.puts n end)

The main trick here is that we keep passing forward a fresh seed. The first one is created in ARandomNumberASecond.generate/1. After that, we keep track of each seed returned from :rand.uniform_s/1 and use that in the next call. Look Ma, no mutable state!

Calling this script will give you a a list that looks something like the following. If your output is exactly the same, you should have bought a lottery ticket this week:

$ elixir a_random_number_a_second.exs
0.8858439425003749
0.8401632592437187
0.22632370083022735
0.8504749565721951
0.22907809547742136
0.302700583368534
0.756432821571183
0.8986912440205949
0.10409795947101291
0.04221231517545583
…

Side Effects

Meanwhile, back in Elm, the evil wizard Random has driven all side effects from the land…

A major goal of functional programming is to limit side effects. We want to be working with as many side effect free, pure functions as possible. Side effects are much harder to reason about and control, so they end up being a major source of bugs in our code.

Elm, in particular, takes this call to arms very seriously. The entire process of seeding and generating random numbers is side effect free. Let’s get into how that works:

$ elm repl
---- elm repl 0.17.0 -----------------------------------------------------------
 :help for help, :exit to exit, more at <https://github.com/elm-lang/elm-repl>
--------------------------------------------------------------------------------
> import Random
> Random.initialSeed 1
Seed { state = State 2 1, next = <function>, split = <function>, range =     <function> }
    : Random.Seed
> Random.initialSeed 1
Seed { state = State 2 1, next = <function>, split = <function>, range =     <function> }
    : Random.Seed
> Random.initialSeed 2
Seed { state = State 3 1, next = <function>, split = <function>, range =     <function> }
    : Random.Seed

You can see here that you need to provide some number when requesting a seed in Elm. If you use the same number, you get the same seed (compare the State numbers). Different numbers yield different seeds.

This makes testing code that makes use of random numbers quite easy. You can create the seed with a known value in tests, pass that into your code, and magically predict the balls as they fall out of the tumbler. That’s a powerful benefit.

A quick aside in defense of Elixir’s honor: it also supports seeding from known values. I just didn’t show it. Throwing Elixir under the bus: you seed it with a tuple of three integers because that’s maximally weird.

Back to Elm. Again.

Elm has another abstraction at play: generators. A generator knows how to build whatever you are after whether that’s integers in some range, booleans, or whatever. They can also be combined, so Random.list could fill a list of a desired size with integers or booleans. Here’s how you build them:

> Random.int -1 1
Generator <function> : Random.Generator Int
> Random.list 10 (Random.int -1 1)
Generator <function> : Random.Generator (List Int)

OK, that’s pretty dull, but we can combine generators and seeds to produce numbers:

> Random.step (Random.int -1 1) (Random.initialSeed 1)
(1,Seed { state = State 80028 40692, next = <function>, split = <function>,     range = <function> })
    : ( Int, Random.Seed )
> Random.step (Random.int -1 1) (Random.initialSeed 1)
(1,Seed { state = State 80028 40692, next = <function>, split = <function>,     range = <function> })
    : ( Int, Random.Seed )
> Random.step (Random.int -1 1) (Random.initialSeed 1)
(1,Seed { state = State 80028 40692, next = <function>, split = <function>,     range = <function> })
    : ( Int, Random.Seed )
> Random.step (Random.int -1 1) (Random.initialSeed 61676)
(0,Seed { state = State 320459915 40692, next = <function>, split = <function>, range = <function> })
    : ( Int, Random.Seed )
> Random.step (Random.list 10 (Random.int -1 1)) (Random.initialSeed 100)
([1,1,0,1,1,-1,-1,1,1,0],Seed { state = State 1625107866 1858572493, next =     <function>, split = <function>, range = <function> })
    : ( List Int, Random.Seed )
> Random.step (Random.list 10 (Random.int -1 1)) (Random.initialSeed 42)
([1,-1,-1,-1,0,-1,1,-1,-1,-1],Seed { state = State 2052659270 1858572493, next = <function>, split = <function>, range = <function> })
    : ( List Int, Random.Seed )

Note that we get generated item and new seed pairs, just like we saw in Elixir. You can also see that the same seed always produces the same value.

That leaves us with just one question, but it’s a big one: where do we get the number needed to create the initial seed? Hardcoding some value works for your tests and this blog post, but doing it for your application proper kind of defeats the point of randomization, am I right?

In my next blog post… OK, OK, we’ll discuss it now. Calm down.

To get an initial seed we’re obviously going to need to do something in a place more tolerant of side effects. It just so happens that Elm code is bootstrapped in JavaScript, where side effects are perfectly legal. We can use JavaScript’s random number generator to kick start random number generation inside of Elm’s side effect free walls.

To show what that process might look like, let’s recreate our number a second streamer in Elm. We begin with this HTML:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">

    <title>A Random Number A Second</title>
  </head>
  <body>
    <script src="ARandomNumberASecond.js" type="text/javascript" charset="utf-8"></script>
    <script type="text/javascript" charset="utf-8">
      Elm.ARandomNumberASecond.fullscreen({
        randSeed: Math.floor(Math.random() * 0xFFFFFFFF)
      });
    </script>
  </body>
</html>

The key bit here is that we pass a program flag into Elm’s invocation. This is a randSeed pulled from JavaScript’s random number generator.

Now we need to dig into the Elm code to see how to make use of that flag. We’ll start with general program concerns:

module ARandomNumberASecond exposing (main)

import Html exposing (..)
import Html.App
import Time exposing (Time, second)
import Random


type alias Flags = {randSeed : Int}


main : Program Flags
main =
  Html.App.programWithFlags
    { init = init
    , update = update
    , view = view
    , subscriptions = subscriptions
    }

This code just imports what we need and sets up execution in version 0.17 style. Note that we do tell Elm that this is a programWithFlags. The model is where we can finally make use of that value:

-- MODEL


type alias Model =
  { currentSeed : Random.Seed
  , numbers : List Int
  }


init : Flags -> (Model, Cmd Msg)
init {randSeed} =
  ( { currentSeed = Random.initialSeed randSeed
    , numbers = [ ]
    }
  , Cmd.none
  )

Elm will pass all flags to your init function. Here that allows us to construct the initial seed. We bootstrap the Model to keep track of two bits of state: the current Random.seed and our List of generated numbers for display.

Here’s the event handling code:

-- UPDATE


type Msg = Tick Time


update : Msg -> Model -> (Model, Cmd Msg)
update _ {currentSeed, numbers} =
  let
    (number, nextSeed) = Random.step (Random.int 1 100) currentSeed
  in
    ( { currentSeed = nextSeed
      , numbers = numbers ++ [number]
      }
    , Cmd.none
    )

We expect to receive Msgs that have the current time inside of them, one each second. We can actually ignore the time, but use its arrival as our cue to append a new Random number onto our list. This code pushes nextSeed forward, just as we did in Elixir.

Next, we need to sign up to receive those TickMsgs:

-- SUBSCRIPTIONS


subscriptions : Model -> Sub Msg
subscriptions _ =
  Time.every second Tick

This code subscribes to receive the Time from the Elm runtime each second, wrapped in a Tick. This passes impure responsibilities on to Elm itself and leaves our code pure.

The final bit is just to render the current list of numbers:

-- VIEW


view : Model -> Html Msg
view {currentSeed, numbers} =
  case numbers of
    [ ] ->
      viewLoading
    _ ->
      viewNumbers numbers


viewLoading : Html Msg
viewLoading =
  text "Loading..."


viewNumbers : List Int -> Html Msg
viewNumbers numbers =
  div [ ] (List.map viewNumber numbers)


viewNumber : Int -> Html Msg
viewNumber n =
  p [ ] [text (toString n)]

That is some very basic HTML generation because this is a programming blog post. Had this been a design blog post, it would have looked gorgeous. Also, it wouldn’t have been written by me.

The only trick in the code above is that we render a "Loading..." message until the first time event shows up about one second into our run. We detect that case by pattern matching on our still empty list of numbers.

You can find the full code for this example in this Gist, if you want to play with it yourself.

When I run the code, my browser begins filling up with numbers:

96

100

17

86

32

60

98

59

97

80

…

It’s worth noting that this isn’t the only way to generate random numbers in Elm 0.17. You can instead generate Cmds that the Elm runtime transforms into events your code receives. This pushes seeding concerns out of your code while still keeping it pure. The documentation has an example of such usage. I’ve used the “manual” approach in this code because it’s harder and I wanted to impress you.

That’s the show, folks. Hopefully you’ve seen how random number generation works in both Elixir and Elm. Maybe you even gained some insight into why this process works the way it does. If you have, please write to me and explain it. Thanks!

James Edward Gray II@JEG2 Engineer at NoRedInk

Running our First Design Sprint

$
0
0

This is a cameo post from our amazing Head of Product, Jocey Karlan!

Make Learning Fun

Learning should be fun. Amidst the many debates raging in the education space, this concept is rarely contested. But while we may agree on the mission of creating inspiring, supportive, and fun learning environments, the method is harder to pin down. Certainly, learning can be fun, but it can also be arduous, tiring, and, at times, frustrating.

At NoRedInk, we believe in mastery-based learning, a paradigm that requires students to prove their understanding of a concept before progressing to the next one. This is drastically different from many traditional models where students move from unit to unit or grade to grade based on a strict timeline. In the world of mastery-based learning, students can’t simply move on after 20 minutes or 20 questions. Rather, they progress at their own pace as they learn.

When we take away easy outs and guaranteed advancement, preserving the fun of learning presents a greater challenge. To make learning fun, we must…

  • Foster a growth mindset and motivate students who are really struggling
  • Develop resources to help students feel supported as they work
  • Celebrate progress and not just completion
  • Provide a delightful visual environment that fosters joy as well as learning

Sprint

Last winter, an advance copy of Sprint arrived from our investors at Google Ventures. Our Product team was drawn to the book’s core premise: carve out 5 days to work on nothing but a single tough problem, condensing a full design-develop-test cycle into one week.

We had a tough problem at hand. For months, teachers and students had voiced feedback about our mastery-based practice engine. We heard regularly from students who felt discouraged or frustrated by progress bars that filled for correct answers and then emptied for mistakes. Instead of fun, for a subset of students, our engine was creating stress. Though we had tried to chip away at developing a solution, we had made little progress.

Thus, Sprint proposed an appealing course of action, and we set out to answer one core question: How might we make students feel like they’re continually making positive progress toward mastery? We called this question “the mastery problem.”

The Team

We put together a cross-departmental team composed of one PM (myself), one developer, and two UX designers.

The Plan

Sprint suggests setting aside 5 neatly slated days. The schedule progresses from mapping out the problem, to sketching solutions, to deciding on a best option, to building a prototype, to finally testing with real users.

The Plan

We used this calendar as inspiration and built a schedule that allowed more time for prototyping and less for decision-making. This was made possible by assigning pre-work to each member of the Sprint team, which included preparing competitor analyses and reading messages from teachers and students.

The Schedule

Day 1:Map and Sketch

  • Lightning talks: Members of our Support, School Partnerships, Product, and Engineering teams gave 15-minute presentations, sharing their context on the mastery problem.
  • Competitor analysis: Each member of the Sprint team presented on 2-3 mastery-based learning platforms.
  • Problem mapping: The Sprint team organized our notes and ideas into various themes (see image below).
  • Sketching: We tried a few sketching activities suggested in Sprint and also allowed for quiet, contained sketch time.

Whiteboard

Day 2: Decide

  • Gallery walk: Each member of the Sprint team posted 1-2 solution sketches around the room. With stickers in hand, we marked the elements of each design that stood out to us.
  • Core themes: We looked for trends in the solution sketches and identified the features that we wanted to implement in our prototype. We trimmed down the list to 6 core themes.
  • Ongoing sketching: We spent another hour sketching and discussing ideas as a group before electing one of our designers to mock up our favorite designs.

Days 3-5: Prototype Here’s where we really “broke process”: Sprint recommends building an extremely low-fidelity prototype that leverages Keynote or prototyping software like InVision. In our case, however, these options wouldn’t cut it. InVision and Keynote are fantastic for testing in controlled environments with a limited number of possible user behaviors. We, in contrast, needed to give kids of widely varying abilities enough freedom to succeed or struggle; only then could we test authentic emotional reactions to our interface.

With exponential possible paths to completing our activity, we opted to engineer a solution (see GIF below). While this solution took a few extra days, we were able to move to largely asynchronous communication to free up team members not immediately involved.

Screenshot

Day 6: Test On the last day of our Sprint, we visited a San Francisco high school and worked 1-on-1 with 8 freshmen students. In case you’ve never done user testing with 14-year-olds, it’s worth noting that their brutal honesty provides data of unmatched quality. These students’ insights and reactions informed many of the changes we implemented after our Sprint.

In summary, here’s how our schedule turned out:

Schedule

What’s Next

In the weeks that followed, we visited one other school to work with students of a different age group and demographics. One of our designers continued to build out various interactions, pairing closely with two developers. Our Product team started to flesh out a spec and brought in a QA analyst to evaluate those initial guidelines.

Conducting our first design Sprint allowed our team to take a long-standing, messy, and emotional problem and come to a solution. Today, we’ve committed to addressing the “mastery problem” in the first quarter of 2017, an undertaking that requires redesigning our entire quiz engine, overhauling old code, and moving student answer data to a new part of our database. While much work remains, we’re confident in developing a solution informed by and built for our users. We can’t wait to release to production.

Like learning itself, creating fun learning experiences for students isn’t easy. But tough problems, a great team, and an inspiring mission are what make being a PM at NoRedInk a delight. Looking for your next role? We’re hiring.

Stay tuned for a follow-up poston how ourmastery problem became our mastery solution.


Jocey Karlan
Head of Product at NoRedInk

Viewing all 193 articles
Browse latest View live