Tuesday, January 3, 2017

Time travelling development--for JS apps

TL;DR To how things should really work, watch this talk by Dan Abramov who created Redux, and the debugger concept.

Long version:
Make a change in your code. See it updated instantly. That's what I want because modern development is an iterative process, with fast iterations.

We've got something like that in JS playgrounds like JSFiddle, CodePen, and so on. But there's still a couple of things that are wrong. Make a change and the app runs again, from the top.  It's pretty quick, and for small bits of code, there may be almost no lag. But as your app scales, the time grows.

You can write a test driver, so that once the app loads, it tests itself, but that's another whole lot of code that you have to write. And even then, reloading takes time, and most of the changes that you make are teeny tiny ones. Even a second or two for a reload is too long.

You can do this in your Dev environment, as well as

And you don't want to test your whole app every time you make a change. Most changes are not going to break the app. They're just little tweaks. What you'd like to do is this: put the app in a certain state. Make your code change. Do some fiddling with the interface. See if it works right. If it doesn't then:

  • Revert to the earlier change
  • Rerun the interface twiddles
  • Repeat until it's perfect
  • Make this the new base, state
  • Repeat forever
And if you run into problems you'd like to go back to your base state with the ability to back up through the twiddles and debug anywhere. Now someone has pretty much done that using mainstream tools.

The key is making your code functional. Take state out of the functions. It's a variation on MVC programming. And it points out a flaw in MVC.

The usual MVC routine is this: you've got a data structure, which is your model. You've got code that takes the Model and renders a View. The view contains UI Control components. When a user manipulates the Controls, the model changes -- either by two-way binding, or by calling functions that are part of the Model, or by directly manipulating the Model data structure. But sometimes the View has its own controlling data structures. So there's a Data Model, and a View Model to pay attention to.

Instead, design code using Model, View, Controller, and Actions. The rules are:

All state is in the Model. Base data state and View state are in different parts of the one and only Model.

Models are POJOs, so they can be serialized and parsed back to their former state.

Actions are Plain Old Javascript Objects (POJOS) emitted by Controllers. 

A part of a Model is updated by calling a model updating function with an Action. There is no other way to update a Model. Model updating functions are called Reducers.

Reducers functions have the following signature: (Oldstate, action) -> newState, the signature used for functions called by Array.reduce They take the prior state as an input parameter and produce a new state. Because of their signature, reducers can be called with an array of Actions, using Array.map. So calling:
arrayOfActions.reduce(reducer, initialState)
Will return the final state, after applying all of the actions. 

Reducers don't update the Model; they replace the old state with a new state, in which the changes have been made according to the Actions. So state objects are immutable.

As a consequence:
  • We can save and restore the application state at any time
  • We can take any state, and apply a set of actions to it, giving the state after the actions
This lets us create a debugger that will do what we want it to do.

The technologies that make this easy to do are React and Redux. React is a framework for view rendering. In ordinary practice, View components can be stateless or stateful. Redux is a framework for managing states for React-based applications so that View components can be stateless.

To see how this works, watch this talk by Dan Abromov who created Redux, and the debugger concept.

Also read the docs for React and Redux, which I am in the process of doing.

Sunday, January 1, 2017

Browser synchronization, event capture

BrowserSync is an Open Source project that keeps multiple browser instances in sync.

I've downloaded it and tested it. It has some problems, but it mainly works as advertised. And it can do a bunch of thing that an awesome environment would need to do.

First: the main idea. BrowerSync captures events from each browser that's displaying data from a site that it serves, and forwards those events to other browsers looking at the same site. Even better, the BrowserSync server can proxy another server, including one serving pages by https.

Here's a review of some competing technologies by Addy Osmani, and his review of BrowserSync or browser-sync (github) and site.

I started by installing browser-sync

npm install -g browser-sync

I started by testing it with a polymer/firebase project. I cd's into the directory and ran:

$firebase serve 

That gave:

Starting Firebase development server...

Project Directory: /home/mwolf/tools/prog-with-papa
Public Directory: dist

Server listening at: http://localhost:5000


In another console, I run browser-sync in proxy mode:

browser-sync start -p  http://localhost:5000

That gives:

[BS] Proxying: http://localhost:5000
[BS] Access URLs:
------------------------------------
      Local: http://localhost:3000
   External: http://192.168.1.2:3000
------------------------------------
         UI: http://localhost:3001
UI External: http://192.168.1.2:3001
------------------------------------

And pops open a window on http://localhost:3000

I opened an incognito window and went to the same URL. Sure enough, clicking on menu selections in one made the same changes happen in the other. One of the pages had a text box. Entering text in one entered it in the other. But scrolling the form did not sync! I also tested it using my phone and the external port. Worked fine.

Next test. I proxy a site on the web.

browser-sync start -p  http://slatestarcodex.com/

Next test. I proxy a site on the web. Works fine, including navigation within the site and scrolling. So maybe the scrolling problem has something to do with Polymer?

Then I proxy the site using https. I get:

[BS] Proxying: https://slatestarcodex.com
[BS] Access URLs:
-------------------------------------
      Local: https://localhost:3000
   External: https://192.168.1.2:3000
-------------------------------------
         UI: http://localhost:3001
UI External: http://192.168.1.2:3001

Again, it works as hoped for.

Will it work for google?

browser-sync start -p  https://www.google.com

Opens a browser, but tells me that there's a security problem: the site doesn't have a correct SSL certificate. I approve it as an exception, and it works!

I try with facebook. Facebook lets me get public content, but its security protocols won't let me log in. I assume that the same thing will happen with any site withy good security.

So BrowserSync does some of the things that I want it to do. Others, like bypassing security, are probably impossible by any means.

Couple this with Chrome Plugins, and I may be able to do some of the hacking that I want to do.

At the heart of browser-sync is its ability to capture events and forward them. On the road to discovery, I found the following resources:

HTMLGoodies Advanced Javascript event handling

Ghostlab

Comparison of Ghostlab and BrowserSync

EventTarget.addEventListener at MDN, defines the basic procedure for handling events. By overriding EventTarget.prototype.addEventListener you can capture events for all targets, and do funky things with them.

An article about event capture on css-tricks.com by one of the authors of GhostLab. The article includes links to several CodePen examples for capture.


The workbench paradigm

IDEs and other power software tools suffer a tension between power and complexity. A photo editing tool has a dozen menus; each might have a dozen selections, and many of those might have submenus. If you remove all the menus and selection, you remove the tool's power. If you keep them, then finding out how to do the thing you want to do next means searching for the right command.

Some tools get you around this by letting you type a string that fuzzy-matches to the function that you are looking for. That's better, but still requires quite a bit of fiddling to get the function that you want.

Instead, consider a shop project as a governing paradigm. A skilled craftsman has a workshop full of tools and materials, but for any one task, needs only a few. Before commencing work the craftsman grabs the needed tools and materials, lays them out on the workbench in easy reach, and then sets to work. When possible, a workman prefers materials as close to finished size as possible. He'd prefer already-turned drawer pulls to blocks of raw wood he can turn into pulls.

So, consider building a GUI. The usual method is to go to a palette of components, pick one, place it, then assign properties to it. Then pick the next one, and so on. Instead, pick only the components that you'll need; choose the items in the palette that you want to be visible, and hide the rest; when you've adapted one to size, add it to the palette.

Consider building something with a bunch of icons using Polymer's iron-icons. If we were doing it with a GUI builder, we might pick iron-icon from a palette of iron elemements (there are 35 of them) place it, pull up a property sheet to set its properties, and continue.

One property is the icon image name from an iron-iconset. So we'd choose the icon images that we want. Then on to pick the next icon. If we're a little smart, then instead of picking another icon from the palette we'd duplicate the first one. But we're still left with the task of choosing another image.

Instead, we'd do better to go to a collection of pre-made iron-icons, each with a different image, and use them as our components.

The Polymer library lets us define a collection, and add iron-icon to the collection. But we can only add the raw iron-icon element's definition. And it has another element that contains a set of icon images. But it doesn't have a nice way to subset the images, or to attach a chosen image to an icon instance other than cut and paste with the text editor.


Saturday, December 31, 2016

Spreadsheet programming

People who don't call themselves programmers can build sophisticated programs using spreadsheet programs. How come? And what implications does it have for building better programming tools?

1. Normal programming deals with control flow. Spreadsheets deal with data flow. That means that sequencing issues disappear. In a procedural language, you have to call a function at the right place in the program: after all its inputs have been set and before its outputs are needed. In a spreadsheet, this happens automatically.
2. Spreadsheets give fast feedback. Some development environments, like JSFiddle, JSBin, CodePen, Plunker, and other javascript playgrounds do this. So do various interpretive languages. But as programs grow, feedback slows. Spreadsheets, on the other hand, give fast feedback to small changes with low impact, slower feedback to changes that have global impact.
3. Spreadsheets know when you're finished with a change. As soon as you move out of a cell, the spreadsheet recalculates (assuming no errors). Playgrounds need to decide whether you're finished editing or not. If they guess wrong, there's a noticeable penalty.
4. Spreadsheets let you organize information multi-dimensionally. Normal programming languages are entirely linear. Spreadsheets give you two dimensions, at least. Within a sheet's two dimensions you can chunk information arbitrarily.
5. Spreadsheets make it easy to visualize information. You can grab key parameters and put them all in one area.
6. Spreadsheets help you see dependencies.
7. Spreadsheet cells are pure functions. There are no side effects.

Of course, there are disadvantages. You can only put so much logic in a cell before you need to resort to a scripting language. And then you have the usual problems of dealing with procedural languages. And you can only transfer a single value--a number or a string. No structures allowed. And the spreadsheet grid limits presentation options. And binding is one-way.

But imagine a spreadsheet-like model that removed some of these limitations. A 2D area might be subdivided into a "model" section and a "view" section. Values in the model would be bound to areas in the view so that, by default, changing the model would change the view, and vice-versa.

Widgets in the view might have property sets that were, themselves, represented by subordinate spreadsheets, and every value in the property set could contain a simple formula.

As the system scales the number of cells grows, unmanageably so we need a way to restrict our view to only those elements of interest at a given moment. 

This starts to look a bit like a standard GUI-building environment, with widgets with property sheets. There are important differences, though. One is: 

Monday, December 29, 2014

Chrome Remote Operation--work in process

Now that I'm making a debugger abstraction it's time to think about eating my own dogfood by connecting my IDE to a Chrome debugger instance running an application--or even another instance of the debugger.

Even better, it looks like the the chrome-remote-debugger API will not only let me debug a remote instance of Chrome, it will let me control and monitor everything that happens.

The node module is here.

To run more than one debugger on an instance requires crmux, here.

And a partially implemented console that I might be able to read to understand what to do is here.

So far I've been able to do this:

1. Run my debugger with an external debug port:

google-chrome --remote-debugging-port=9222&
9222 is the standard debugging port.

2. Run crmux in a separate window

crmux

This multiplexes the debugger on 9222 to be available on 9223

3. Now I can use the chrome-remote-api REPL

./node_modules/chrome-remote-interface/bin/repl.js -p 9223
Proof that it works: if crmux is not running, it gives an error.

4. Now I can open a page at

     http://localhost:9223

This gives me a list of debuggable pages. If I choose the one that I want I get this URL:

http://localhost:9223/devtools/devtools.html?ws=localhost:9222/devtools/page/7F642E19-7F9A-40AB-BE87-B344B6307246

Note the ws=localhost:9222. This is the original port and can interfere with things. So I must manually change to ....?ws=localhost:9223...

Proof that this works: I can kill crmux and things no longer work.

5. If I now run

crconsole -p 9223

I get a prompt. (It fails if crmux is not running)

And if I type the command

.tabs

I get something like this:

localhost> .tabs
[0] http://localhost:9223/devtools/devtools.html?ws=localhost:9223/devtools/page/7F642E19-7F9A-40AB-BE87-B344B6307246
[1] http://localhost:3333/IDE
[2] chrome-extension://jnihajbhpnppcggbcgedagnkighmdlei/_generated_background_page.html
[3] chrome-extension://ighdmehidhipcmcojjgiloacoafjmpfk/_generated_background_page.html
[4] chrome-extension://diebikgmpmeppiilkaijjbdgciafajmg/background.html
localhost> 

OK. Those are the right tabs.

But no matter what order I use, I haven't been able to get both the Chrome debugger and crconsole ot play nicely together.

But: I was able to run an instance of crdebug and an instance of the chrome-remote-interface REPL together, and evaluate the following this in the REPL

chrome> Runtime.evaluate({expression: 'foo = 20'}, console.log)
Output is:
chrome> false { result: { type: 'number', value: 20, description: '20' },  wasThrown: false }
And the result, 'foo=20' appears in the other console. So I have two different consoles talking to the same chrome instance.

The problem comes when I run one chrome debugger instance along with a crdebug instance. Depending on which one I start first, the other fails.

If I run devtools first, then crconsole fails:
TypeError: Parameter 'url' must be a string, not undefined    at Url.parse (url.js:107:11)    at Object.urlParse [as parse] (url.js:101:5)    at WebSocket.initAsClient (/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crconsole/node_modules/chrome-remote-interface/node_modules/ws/lib/WebSocket.js:475:23)    at new WebSocket (/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crconsole/node_modules/chrome-remote-interface/node_modules/ws/lib/WebSocket.js:59:18)    at Chrome.connectToWebSocket (/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crconsole/node_modules/chrome-remote-interface/lib/chrome.js:112:15)    at Object.ChromeREPL.setTab (/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crconsole/index.js:223:17)    at Object.<anonymous> (/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crconsole/index.js:23:12)    at /home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crconsole/index.js:167:7    at IncomingMessage.<anonymous> (/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crconsole/node_modules/chrome-remote-interface/lib/chrome.js:104:13)    at IncomingMessage.EventEmitter.emit (events.js:117:20)
The code c

Then crmux fails this way:
/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crmux/crmux.js:103           msgObj.id = idMap.id;                            ^TypeError: Cannot read property 'id' of undefined    at WebSocket.<anonymous> (/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crmux/crmux.js:103:29)    at WebSocket.EventEmitter.emit (events.js:98:17)    at Receiver.self._receiver.ontext (/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crmux/node_modules/ws/lib/WebSocket.js:682:10)    at Receiver.opcodes.1.finish (/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crmux/node_modules/ws/lib/Receiver.js:391:14)    at Receiver.expectHandler (/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crmux/node_modules/ws/lib/Receiver.js:378:31)    at Receiver.add (/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crmux/node_modules/ws/lib/Receiver.js:87:24)    at Socket.firstHandler (/home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crmux/node_modules/ws/lib/WebSocket.js:663:22)    at Socket.EventEmitter.emit (events.js:95:17)    at Socket.<anonymous> (_stream_readable.js:746:14)    at Socket.EventEmitter.emit (events.js:92:17)
The crmux code can be debugged with:

node-debug --preload 0 /home/awesome/tools/node-v0.10.26-linux-x86/lib/node_modules/crmux/crmux.js

The problem occurs when we can't map an upstream ID to a downstream one. That is going to take a bit before we can deal with it. It probably has to do with the DevTools debugger making sure that the IDs that it gets are correct ones, and that it does not see an ID that it does not understand.

I can probably solve this by opening a debugger on the debugger and debugging it.


Tuesday, July 8, 2014

Interface to Coffeescript debugger

```node-inspector``` is the way to start a debugger, along with a script to compile coffee to js. There are several problems with the way things work.

The ideal: I've got a testServer that automatically runs tests when either the code or the test is changed. I'd like the following to happen:
1. If I add a statement, like ```debugger```,  to the code then instead of running the program, it runs it in the debugger.
2. When the debugger runs, instead of going to the first line of code it runs to the statement.

Right now I can't do that. Node inspector either stops on the first line of code or it runs until it gets a USER KILL signal, and then stops on a ```debugger``` statement, or must be stopped manually.

If I use a ```debugger``` statement then every time the statement is hit, the debugger stops.

I could solve this by having my own debugger, and ultimately that might be the way that I want to go.

This issue https://github.com/node-inspector/node-inspector/issues/240 describes how ```node-debug``` might be modified to start the process running as soon as it comes up. It also points to the critical code for debugging.


Sunday, July 6, 2014

CoffeeScript Source maps

CoffeeScript has its virtues. It's terse and expressive. And using coffeescript/register you can tell node to load your coffeescript files instead of looking for javascript.

But there are problems. Source maps make it easy to debug, but to use source maps you must first compile the code to javascript. This clutters up the file system.

There are other problems as well. If you configure mocha to use coffeescript tests then mocha will load the coffeescript without debug information, which means you're back to using javascript and your stack traces will give javascript lines instead of coffeescript lines.

There are solutions, but they're going to be a bit of work. The coffeescript compiler has an option, "inline" that generates source maps along with compiled javascript. The source maps can be appended to the javascript, and everything should work right.

The inline-source-map project (https://github.com/thlorenz/inline-source-map) shows how this can be done.

The coffee-inline-source-map project (https://github.com/thlorenz/inline-source-map) compiles coffeescript with inline maps.

Source maps are created by http://coffeescript.org/documentation/docs/sourcemap.html

Stack traces for coffee files are created by Error.prepareStackTrace, found in http://coffeescript.org/documentation/docs/coffee-script.html