How Haskell can solve the integration problem

Will show how long running tasks, Web apps, workflows, EAI Orchestration and BPM applications share the same underlying problem: The "integration problem", that only haskell can solve with more simplicity, generality and maintainability

Creating a single application in the imperative style is easy and intuitive, because the programmer is in control of the sequence of things to do. But when it comes the time to integrate two or more autonomous entities that send events at any time in its own sequence then is when the programmer is not in control, so a different programming model is necessary. Such problem happens when trying to integrate the users with backoffice applications via web applications, but also when it is necessary to integrate two or more backoffice applications, company departments, web sites, web services etc.

The standard model that solves this inversion of control problem has various names but esenstially is the same architecture with different names: finite state machine, state-transition system, a state machine system or a event handling model. That is the architecture of the main web frameworks, Enterprise Application Integration (EAI) frameworks, Orchestration frameworks, Workflow frameworks, Service Oriented Architecture (SOA) frameworks and Business Process Management (BPM) frameworks, that solve respectively, the individual above mentioned integration problems.

See the tutorial at

https://www.fpcomplete.com/user/agocorona/how-haskell-can-solve-the-integration-problem

With a practical example.

10+ things that you can do with MFlow and you can't with your Web framework

1. Create test, integrate and install your logic without concern for layout. Edit your forms, widgets, style and content at run time.
2. Convert your application from single page to multiple page and back with little code modifications
3. Make forms that change their questions depending on your answers.
4. Make a cascade menu with dynamic options programmatically in a single procedure
5. Make an element of a page to refresh itself by adding a single statement
6. Make an element to push its content with a simple modifier
7. Press back as many times as you like by default
8. Write a payment flow with some pages and seamlessly drop it whenever you need it
9. Write an active page element with his own server code, JS, CSS in a single procedure and seamlessly drop it whenever you need it
10. Write your routes and control logic as in a console application. No spaguetty callback code
11. Transparently store and retrieve session data for as long as you wish
12. Make all of this without writing a single line of javaScript code. Although you can add it.
13. Make all of this in a type safe way. If your app compiles, it works.
14. Make (almost) all of this work with or without javascript activated.
15. Make all of this in an architecture that is horizontally scalable (although not implemented such scalabiltiy yet)

Quick Start : Basics for understanding and using MFlow

Quick Start 2 : how to modify an application to add dynamic effects: implicit ajax, push etc.

MFlow site:  http://mflowdemo.herokuapp.com

Learning from the Haskell compiler

Haskell programming is naturally one level of abstraction above other languages either functional or not. It is not the functional paradigm. It is the type inference and, specially, the category theoretical grounds in which the language is based, for which functional and imperative paradigms are particular cases.

The extra level of abstraction means that you are one level up in the tree of possibilities. For example, you can define the very meaning of a sequence of statements in a Monad Instance. That, in other languages is fixed.

That means that you have a extra level of things to learn and explore. For example I created a DSL for web applications time ago and still I´m learning my own language. Sometimes my programs type check, but they do not perform what I was expecting, But it does other interesting thing. The code was a product of my own misunderstanding or my lack of concentration, but the type checker say: its valid, let´s run it. Then I check it again and voila, when I examine the program I discover a new usage of the same elements I defined that I never though previously possible.

Sometimes the error messages tell me new instances of things that I would never though about. Some of them seem crazy, but after a second though, I find that is not as crazy as it seems but, sometimes I´m not willing to accept yet another extra level of abstraction that my compiler is suggesting as result of an error.

For beginners this flood of information is confusing,as well as for expert programmers they just want their job done, but thanks in part to this interaction I´m here talking about abstract web combinators when what I initially needed from Haskell was the fsstest way to see "hello world" in a web browser.

More composable elements for single page development: witerate and dField

witerate,  is a new primitive that permits to iterate the presentation of data and/or input fields and widgets within an web page that does not change. The placeholders are created with dField.  Both are widget modifiers: The latter gets a widget and create a placeholder in the page that is updated via ajax. The content of the update is the rendering of the widget at each iteration. The former gets a widget which contains dField elements and permit the iteration. Whenever a link or a form within the witerate widget is activated, the result is the placeholders filled with the new  html content.  This content can be data, a input field, a link or a widget. No navigation happens.

This permits even faster updates than autoRefresh.  since the latter refresh the whole widget and it does not permits modifications of the layout at runtime. When edTemplate or template is used on top of witerate, the result is editable at runtime, and the span placeholders generated, that are updated via ajax can be relocated within the layout of the template.

Additionally, contrary to some javascript frameworks, the pages generated with this mechanism are searchable by web crawlers.

This example below, taken from the runtime templates example, shows how template, witerate and dField work together. The example iterates the presentation of a list of results fby displaying four of them each time. The list can be navigated forward and backward. (see the example  running and the full source code here).

These are two pages of the example with the templates beind edited at runtime. The first present results and the second is an input form managed the same way. Since the pages are not refreshed, this permits very fast input and presentation of results.

The full post explaining everything is here:

http://mflowdemo.herokuapp.com/noscript/wiki/singlepage.html

See the page and the full source code here

About client-side frameworks

Before any consideration about either client-side or server side framworks are good for one or other user case, you, as a programmer has to answer a simple question: is the Web browser a good development, integration, test and exploitation platform? My response is: No.

I love JavaScript. I programmed AJAX applications using hidden frames 13 ago, before this technology had a name. I regularly used client side technologies for editing documents, for example, Google Docs. But what makes great Google docs is the server integration of many client and server side developments.  

You finally must integrate different aspects of your applications: user data, different client-side applications , snippets to compose workflows and you have to decide where you integrate all of this.  Either you integrate it in the server of in the client.  if you do it in the client, you hardly will do multi-user stuff. you will not have a type safe, fast, multi-threaded environment under your control. You are restricted to the single threaded, single user javaScript environment. if you use some framework above that you still are restricted to the limitations of the JavaScript virtual machine and development environment. 

There are lots of client side frameworks that promise a declarative extension of HTML, but if you look at AngularJS, you will see a lot of JavaScript code, with all the big complexities of big server side frameworks, but, looking at the examples, these frameworks are devoted to present server data, in which they add little  additional advantages, except perhaps less server-client traffic. A server side framework can do it as well with adequate use of AJAX, (see this)

To summarize you have the problems of two tier architectures, the thick client problem aggravated with less power of the sandboxed environment of the web browser. Until you load all the client framework, the first page display can experiment delays. Although this could not be a problem in intranet environment where two tiered architectures have been good for a limited number of users, that is specially critical in applications with a casual usage pattern, such are all Web applications.  The dynamic HTML is not searchable. You must see how Twitter had to revert to the server side.

You may think that a server with application logic and a MVC event driven logic in the middle tier, can solve the problem, but, simple speaking, the business applications are not stateless. Neither you can store the state in the client allways. Have you seen a flowchart?, a user requirement document? all of them are filled with steps. Specially in a corporate environment, where it is necessary to integrate different software elements, departments, databases etc, much of the integration of different modules with the corresponding interfaces must be trough an stateful application that present different interfaces.

The question is not either client or server side. Both parts must believe that they are the center of the development. the user too. When you are programming in the client, you must think client centric. when you are in the server, you have to feel server centric. The intelligent way is  make everyone: The user, the front-end programmer, the back-end programmer, the integrator, to feel that they are the center of the development.

And the development process in which each one works at the highest level of productivity and abstraction is when you can package server and javascript functionalities that work together in easily pluggable, self contained widgets for the user interface. With these widgets, let the application programmer to compose single-page applications and finally with these single-page applications, integrate them in stateful workflows that accomplish the whole user case.  That is the aim of MFlow.

Change the content & layout of an active page with WYSIWYG at runtime?

It is possible modify the layout of the active components at run-time. This means that no longer is necessary to code a layout for a formulary,  or for the arrangement of different widgets. 

Just create them without layout, and later the people in charge of the layout and styles will arrange the layout and the texts when the application is already tested. Then the layout never pollutes the code, and it may be decoupled also in time. 

The layout can be edited in a more powerful editor and inserted again and so on. As long as the designer do not modify the tags of forms and links created by the application, everithing goes fine. It can insert wathever content, formatting, apply styles etc.

The validation errors in the formularies must be presented via Javascript however. The example uses a simple javascript alert to present an error when the message entered in the form field exceed a certain length

http://mflowdemo.herokuapp.com/noscript/templates/runtimetemplates

The example list the names entered in a input form, but at the same time it permits the edition of the three pages: The first page has links. There is a page with a form for entering a new name and the content of a list of results.It uses edTemplate and edTemplateList, two new primitives for template edition.

Log in as edituser/edituser to edit the pages using the login option. Log out to see the resulting layout. The edited layour has additional information about how it works.

A cascade menu coded in pure applicative & monadic haskell?

Now the mflow demo has a cascade menu.

• Database examples with different backends
  • Persistent illustrates the use of MFlow with Persistent(In this example sqlite backend is used) (article)
  • Database Create, Store and retrieve lines of text from Amazon SimpleDB storage (article)
• Push Samples using long polling
• Error Traces
• Different kinds of flows
• Basic Widgets
• Monadic widgets, actions and callbacks autoRefresh, page flows, dialogs etc
• Dynamic Widgets Widgets with Ajax and containers of other widgets
• Runtime templates Templates and content management modifiable at runtime
• Login/logout

The menu works as any cascade menu, but it does not use explicit javascript to display and hide the branches when clicked. It uses pure haskell with applicative and monadic combinators.

It uses the behaviour of the monad instance of the  MFlow  formlets. But in this case it uses links instead of forms elements.

Here is the code of two branches of the cascade menu:

absLink ref = wcached (show ref) 0 . wlink ref

mainMenu ∷  View Html IO Options
mainMenu= autoRefresh $
  ul <<< li <<<  do
          absLink DatabaseSamples << b  "Database examples"
             <++ " with different backends"
          ul <<<
           (li <<< (absLink MFlowPersist <<  b "Persistent")  <! noAutoRefresh
                     <++ do -- blaze-html monad
                        b " illustrates the use of MFlow with "
                        a  "Persistent" ! href yesodweb
                        " (In this example sqlite backend is used) "
                        article persistentarticle

           <|> li <<< (absLink Database << b  "Database") <! noAutoRefresh
                     <++ b " Create, Store and retrieve lines of text from Amazon SimpleDB \
                            λ storage "
                     <> article amazonarticle)
   <|> li <<<  do
          absLink PushSamples << b  "Push Samples"
             <++ " using long polling"
          ul <<<
           (li <<< (absLink Push << b  "Push example") <! noAutoRefresh
                     <++ b " A push widget in append mode receives input from \
                             λa text box with autorefresh"
                     <> article pushl
                     

           <|>   li <<< (absLink PushDec << b  "A push counter") <! noAutoRefresh
                     <++ b " Show a countdown. Then goes to the main menu"
                     <> article pushdec)
   <|> li <<< do
      ......

It works as follows:

The menu is a set of links composed by applicative alternative (<|>) operators. each operator contains a do sequence  so according with the monad instance, the sequence executes as long as the previous lines of the sequence are validated by the user input, so when the first link in the do sequence is clicked, the next sentence can be executed. So when the top level link is clicked, what appears is a second level of links composed again with the (<|>) operator. They are the terminal branches in this case. (the process can go down further with a new level, if it is necessary). The final branches are absolute links with the class= _noAutoRefresh attribute. This tell autoRefresh to escape from local auto-refreshing and permit a page navigation when this link is pressed.

autoRefresh install the javascript and server flags necessary for auto-refreshing the menu without refreshing the whole page.

The operator (<<<) encloses the HTML of a widget within anoter HTML tag. In this case the tags UL and LI encloses the links.

absLink is a cached wlink and thus has a fixed path. this absolute link is convenient since the code backtrack when a menu item is cliked from any option (see this article).

 (<++) append HTML code to a widget.

The behaviour can be improved, for example,if  instead of collapsing each branch when a new branch is open, to maintain any number of branches open,  just add its own autoRefresh for each branch instead of a general autoRefresh.


One additional NOTE about this code: There is a round-trip to the server each time an option is pressed, but for once, this permits the presentation of mutable data which is arranged in three-like structures, not only static menus. In the other side, the server code that respond the requests is ready waiting and with all the context in place, no lookups, typical of MVC frameworks are necessary, so the response has a very short latency, about 5 ms latency in the server, according with my heroku box.  Here are the typical response parameters for the menu refresh:

 connect=1ms service=5ms status=200 bytes=2862

What I have been doing: Runtime templates and Content Management

Hi,

There are two exciting things that I´m being doing lately: Content management has been improved a lot. Now the editors can edit large texts at runtime and see the result instantly. They have an edition panel with all the ordinary functionalities including uploading and linking images, edition of raw HTML and so on. Different users can have permissions for different sections. See the content of the home page of the MFlow demos. It has been created with this content management facility at runtime. The content management demo shows how it works.

After you have logged with the right user, to edit the text click on them. immediately, the panel above is activated and the content can be edited. At the end, clicking the save button, the edition facility and the panel disappear and the editor see the exact layout of the page. To edit it again, just refresh the page. The content is published immediately. for the public. The content is saved in the texts folder.

Very related with this, but more exciting and powerful, is the possibility to modify the layout of the active components at run-time. This means that no longer is necessary to define a layout before compilation for a formulary,  or for the arrangement of different widgets. Just create them without layout, and later the stylist will arrange the layout and the texts when the application is already tested. Then the layout never pollutes the code, and it may be decoupled also in time. he layout can be edited in a more powerful editor and inserted again and so on. As long as the designer do not modify the tags of forms and links created by the application, everithing goes fine. It can insert wathever content, formatting, apply styles etc.

This does not end here. There may be more than one layout,with more or less text, advertising etc depending o the device. For example,  a slim layout for mobile phones, other more heavy for PCs etc. This is something that I´m just developing now. T

MFlow using persistent with sqlite backend

There are excellent libraries that I want to make interoperable with MFlow one of them is Persistent. It allows to define backend-independent data layers. In the previous post I mentioned an example of direct integration of TCache -the cache of MFlow- with Amazon web services. In this example I will show how MFlow and Persistent directly interact. 

Since in this case, the database is a in memory instance of sqlite, It does not make sense to cache data. But since Persistent can interact with remote SQL and nonSQL databases, I need to integrate TCache with Persistent for caching in the near future.

But first, this example illustrates the use of MFlow with Persistent. The example is at:

http://mflowdemo.herokuapp.com/noscript/mflowpersist

The code is taken from http://www.yesodweb.com/book/persistent by modifying the first example and making some guesses by looking at the prostgresSQL version , both are console-oriented programs.

Note how little additions are necessary to change a console application of the sample to a MFlow application.

The example has a navigation of four pages and you can go forward and backward. While the flow looks like an ordinary imperative program, yo can go back and fort and to introduce any bookmark without producing navigation errors.

Additionally you can press the back button, change the form input and see how the responses match the register values. 

This is the code:

{-# LANGUAGE EmptyDataDecls    #-}
{-# LANGUAGE FlexibleContexts  #-}
{-# LANGUAGE GADTs             #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE QuasiQuotes       #-}
{-# LANGUAGE TemplateHaskell   #-}
{-# LANGUAGE TypeFamilies      #-}
{-# LANGUAGE NoMonomorphismRestriction #-}

module MFlowPersistent

where

import           MFlow.Wai.Blaze.Html.All
import           Control.Monad.IO.Class  (liftIO)
import           Database.Persist
import           Database.Persist.Sqlite
import           Database.Persist.TH

import           System.IO.Unsafe

import           Menu



share [mkPersist sqlSettings, mkMigrate "migrateAll"] [persistLowerCase|
Person
    name String
    age Int Maybe
    deriving Show
BlogPost
    title String
    authorId PersonId
    deriving Show
|]

-- Uncomment this to run the example alone
--main= do
--  migratesqlite
--  runNavigation "" . transientNav $ mFlowPersistent
-- 
--askm= ask 

  
pool= unsafePerformIO $ createSqlitePool ":memory:" 10

runSQL sql= liftIO $  runSqlPersistMPool sql pool

migratesqlite= runSQL $ runMigration migrateAll

mFlowPersistent ∷  FlowM Html IO ()
mFlowPersistent = do
    (name, age) ← askm $ (,) 
                         <$> getString Nothing <! hint "your name" 
                         <++ br
                         <*> getInt    Nothing <! hint "your age"
                         <** br
                         ++> submitButton "enter"
                         
    userId ←  runSQL  $ insert $ Person name $ Just age

    post ←  askm $ getString Nothing <! hint "your post" <** submitButton "enter"
    runSQL  $ insert $ BlogPost post userId

    oneUserPost ←  runSQL  $ selectList [BlogPostAuthorId ==. userId] [LimitTo 1]

    askm $   b << show (oneUserPost ∷  [Entity BlogPost]) 
         ++> br 
         ++> wlink () << b  "click here"

    user ←  runSQL  $ get userId

    askm $   b << show (user ∷  Maybe Person) 
         ++> br ++> wlink ()  << b  "click here"
    where
    hint h=  [("placeholder",h)]

Using Amazon Web Services with TCache and MFlow

The code of this example is identical to any one for persistence in files that you can find here except the procedure  setAmazonSimpleDB that set the default persistence in Amazon simpleDB. 

Here is the example running:

http://mflowdemo.herokuapp.com/noscript/database

As ever, MFlow permits to address all the pages in the flow with a single bookmark. You just enter the appropriate path and parameters. For example this URL:

http://mflowdemo.herokuapp.com/noscript/database/newtext

Present the entry form, that is the second page in the demo. And this URL:

http://mflowdemo.herokuapp.com/noscript/database/newtext?p0= text to enter

Fill the text box of the entry form, insert the text in the database and return to the first page again, since the demo has a loop.

Default persistence means that each register is stored in a small blob somewhere, either in a file -by default- or in another storage. For small blobs identified by keys,  Amazon simpleDB is better than  S3. 

The default persistence is defined in the setDefaultPersist and setPersist. While the former -as its name says- set the default for any kind of data, this latter is per-datatype. It is a method defined in the Serializable instance in the DefaultPersistence module. The blob of each register is read and written by TCache and converted to a haskell register under a  DBRef reference, in the STM monad. DBRefs follows the semantics of the TVar references.

TCache besides caching the registers, it permits querying and indexing them by a relational-like syntax using haskell register field names (see the query defined below in allTexts). The indexes used for querying are also registers that are stored and retrieved using the default persist mechanism.  That means that the registers that are not used are discarded from the cache, and the modified indexes are stored in the persistent storage automatically, in this case, in SimpleDB

Caching and querying local indexes improves response time and reduces the number of accesses to the paid cloud infrastructure, so it reduces costs.

I´m not fully satisfied with the default persistence set in setAmazonSimpleDB  I need to optimize the storage and the access using more features of  AWS simpleDB and mix it with Amazon S3 for largue blogs, but this is a proof of concept.

This is the code:
{-# LANGUAGE DeriveDataTypeable, RecordWildCards #-}
module Database where
import MFlow.Wai.Blaze.Html.All hiding (select)

import Data.Typeable
import Data.TCache.IdexQuery
import Data.TCache.DefaultPersistence
import Data.TCache.Memoization
import Data.Monoid
import Menu 
import Data.String
import Aws
import Aws.SimpleDb hiding (select)
import qualified Data.Text as T
import Data.Text.Encoding
import Data.ByteString.Lazy(toChunks,fromChunks)
import Network

import Debug.Trace
(!>)= flip trace

data  MyData= MyData{idnumber ∷ Int, textdata ∷  T.Text} deriving (Typeable, Read, Show)      -- that is enough for blob persistence

instance Indexable MyData where  
   key=  show . idnumber    -- to notify what is the key of the register
   defPath = const ""
   
data Options= NewText  | Exit deriving (Show, Typeable) 


main= do
  setAmazonSimpleDB
  syncWrite  $ Asyncronous 120 defaultCheck  1000
  index idnumber
  runNavigation "" $ transientNav $ do
     all ←  allTexts
     r ←  ask $ listtexts all
     case r of
         NewText → do
              text ←  ask $   p << "insert the text" 
                           ++> getMultilineText "" <++ br
                           <** submitButton "enter"

              -- store the name in the cache 
              -- (later will be written to disk automatically)
              liftIO . atomically . newDBRef $ MyData (length all) text 

         Exit → return ()
     where 

     menu=   wlink NewText   << p << "enter a new text" <|> 
             wlink Exit      << p << "exit to the main menu"
               
     listtexts all  =  do
           h3 << "list of all texts"
           ++> mconcat[p <<  t | t ←  all]
           ++> menu
           <++ b << "or the back button for a new database action"

     allTexts= liftIO . atomically . select textdata $ idnumber .>=. (0 ∷  Int) 
     
sdbCfg =  defServiceConfig

domain = fromString "mflowdemotest"

setAmazonSimpleDB = withSocketsDo $ do
 cfg ←  baseConfiguration
-- simpleAws cfg sdbCfg $ deleteDomain domain 
 simpleAws cfg sdbCfg $ createDomain domain  -- delete once created
 setDefaultPersist $ Persist{
   readByKey= λkey → withSocketsDo $ do
       r ←  simpleAws cfg sdbCfg $ getAttributes (T.pack key) domain 
       case r of
        GetAttributesResponse [ForAttribute _ text] → return $ Just   
             $ fromChunks [encodeUtf8 text]
        _ → return Nothing,
        
   write= λkey str → withSocketsDo $ do
       simpleAws cfg sdbCfg 
                     $ putAttributes  (T.pack key)  
                       [ForAttribute tdata 
                        (SetAttribute 
                          (T.concat $ map decodeUtf8 $ toChunks str)
                             True)] domain
       return (),
   delete= λkey    → withSocketsDo $ do
     simpleAws cfg sdbCfg 
     $ deleteAttributes (T.pack key)  
       [ForAttribute tdata DeleteAttribute] domain
     return ()
     }


tdata= fromString "textdata"

How to use backtracking to present the main menu on every page for free

Still I´m exploring the possibilities of expressing the navigation of a web site by means of matching and backtracking. Using a monad with backtracking, and link + form parameters as the elements for the matching mechanism. That is how MFlow handles the web navigation. I increasingly find that this is the right paradigm that will be the standard if future web development. Event handling coding and all their abstruse configurations and hacks for proper state managemnt will be contemplated as the legacy of an ominous past  that the Humanity had to travel before finding the freedom-under-control of the monadic utopia  ;)       

As an example of how natural is the transformation of web navigation requirements in terms of tracking and backtracking in a monadic, imperative-like code, I show you how I solved my last requirement:

I want to have the menu present in every page, in my demo at:  

http://mflowdemo.herokuapp.com

Now it has the main menu available for every page. Additionally, all the examples, including the persistent one (the shopping cart) are in a single flow.

I could have done thr first by just adding the menu as a widget more in each page, but this means that my code has to check for clicks in the menu on every page, besides the concrete code that I´m focused on.

So, if i have this code

r ←  page pagecode            (1)
normalAppflow r

To add a menu to each page I have to change the code in a way that look like:

r ←  ask $ fmap Left menu <|> fmap Right pagecode
case r of
   Right x        → normalAppflow x
   Left  menuitem → processitem menuitem

Or alternatively:
r ←  ask $ menu `waction` processItem **> pagecode
normalAppflow r


With:

processitem item= case item of
                   item1 -> ...
                   ...

In both cases, the menu items would be executed recursively. That is not good for the memory usage of the application since, to allow backtracking, the Flow monad is not tail recursive. The memory is freed when the timeout expires, but still it is not the best solution. It would be better backtrack to the menu before processing the option, so the data of this abandoned branch would be garbage collected.

Alternatively, I can put each demo in a page flow, where each page in the demo becomes a auto-refreshed widget under a single main page together with the menu, but that denaturalize some demos that are inherently made for page navigation. Additionally I don´t want to use such advanced thing as a page flows and auto-refreshing in the home page of a demo that I want to keep as simple and understandable as possible.

So I tried to use the backtracking mechanism in a way that when an item in the menu is clicked in any demo page, instead of checking for it and call again the menu, It backtracks to the menu page, where the flow will track the appropriate branch of execution depending on the menu item chosen.

Now I want not to code this manually, so instead I make my menu tell ask that he is some pages back and will care for the response, so page must initiate a backtraking. This is done with retry:

retry w= w >> modify (\st -> st{inSync=False})

inSync is an internal state parameter. It means that the server is in sync with the browser because the server found a parameter or link sent by the browser that match with the page that the server is now processing. because ask/page is forced to False by retry, he initiates a backtracking until some previous page match the web browser response.

Now the code becomes:

r ←  ask $ retry menu **> pagecode
normalAppflow r

or

r ←  pagem pagecode
normalAppflow r 

which is almost the same than the original code in (1).

With:

pagem pagecode= ask $ retry menu **> pagecode 

The **> operator is the applicative *> but the first ever executes the second parameter no matter if the first succeeded or not. Since all my pages use the same menu, then I can substitute ask and page by pagem, that knows implicitly about the menu. With this exception I have nothing more to change in my application. if no link of form of pagecode is clicked, then page will find itself not in sync, but actually, there have been a request for a link/form in the menu that will be handled by the menu page, back and down in the execution tree. That is why retry is called as such.

A page can have as many retried widgets as you like. It is important to have the retried widgets cached, since cached widgets maintain the parameter numbering for the web forms and the link deep appropriate for the place where they backtrack. Moreover a menu used in many pages is an inherent candidate for being cached, for performance reasons. I though about making these requirement explicit in the type system, but at this moment I find this alternative a bit overengineered.

This is how the navigation monad example would look like with these modifications. The essential changes are in bold:

import MFlow.Wai.Blaze.Html.All
        
main= runNavigation "" . transientNav $ do
  option ←  ask  menu1 
  case option of
    "1" → do
           pagem $ wlink "2" << contentFor "1"
           pagem $ wlink "3" << contentFor "2"
           pagem $ wlink "4" << contentFor "3"

    "a" → do
           pagem $ wlink "b" << contentFor "a"
           pagem $ wlink "c" << contentFor "b"
           pagem $ wlink "d" << contentFor "c"

  pagem $ wlink ()  << p << "back to the first page"

menu1 =  wcached "menu" 0 $ wlink "a" << b << "letters " <++ i << "or "   <|> wlink "1" << b << "numbers"

pagem  pagecode =page $ retry menu1 **> pagecode
     
contentFor x= do
        p << "page for"
        b << x
        p << "goto next page"

header1= html . body

With this code, every page has the menu on the top (letters or numbers). At any page the user can change from letters to numbers by clicking the menu.

Maxwell Smart push counter

To illustrate the push functionality the previous example is confusing since it seems to be a request response interaction with a web browser, but it is not, since the text box send to the server a string, it is stored in a variable, the push widget detect the update of the variable and send it back in a separate message, different that the reception message. In fact various unrelated push widgets can be updated using this asynchronous mechanism   simultaneously in the same page.

So I created another push example theoretically more simple.

http://mflowdemo.herokuapp.com/noscript/pushdec

It is a countdown after which the page will navigate back to the main menu. In this case the widget generates its own output by decreasing the variable. It also illustrates the use of Hamlet.

Note that

 - push has a new parameter that is the delay for a new ajax request when the previous connection has been lost.

- The last push after 0, send a script that forces a navigation to the menu. And then kill itself.

pushDecrease= do
 tv ←  liftIO $ newTVarIO 10
 page $ 
  [shamlet|
   <div>
       <h2> Maxwell Smart push counter
       <p> This example shows a reverse counter
       <p> To avoid unnecessary load, the push process will be killed when reaching 0
       <p> The last push message will be an script that will redirect to the menu"
       <h3> This message will be autodestroyed within ‥

  |] ++>  counter tv <++  b << "seconds" 
 where

 counter tv = push Html 0 $ do
      setTimeouts 100 0     -- kill  the thread if the user navigate away
      n ←  atomic $ readTVar tv
      if (n≡ -1) 
        then  do
          script << "window.location='/'" ++> noWidget
          liftIO $ myThreadId ↠ killThread 
        else do
          atomic $ writeTVar tv $ n - 1
          liftIO $ threadDelay 1000000
          h1 << (show n) ++> noWidget

atomic= liftIO . atomically

.....

Ops.. No.

killThread is never called since noWidget return an invalid formlet value, so the next statements are not executed in the View monad (See: The promising land of monadic formlets). 

That kill is in order to drop the process immediately, but otherwise, the timeout of setTimeouts would do the same job. Since the delay between push sends is one second, by shortening the timeout from 100 to two seconds will do the same job.  So the killthread line can be eliminated.

New push mode, to present data in real time. in MFlow

Another milestone completed in MFlow.  The new asynchronous widgets can display server information at the moment that they appear. And thanks to Software Transactional Memory this may be immediately.

The new primitive, push, like autoRefresh,  is a modifier of a widget behavior. In this case push  will execute the widget and present the output again and again using ajax internally. This is an example:

http://mflowdemo.herokuapp.com/noscript/push

 

 

The image is not very exciting. I will try to add another better. In this example the push widget is above the text box.

 

The code is as follows:

 

pushSample=  do
  tv ←  liftIO $ newTVarIO $ Just "The content will be appended here"
  page $   h2 << "Push example"
       ++> p << "The content of the text box will be appended to the push widget below."
       ++> p << "A push widget can have links and form fields."
       ++> p << "Since they are asynchronous the communucation must be trough..."
       ++> p << "The input box is configured with autoRefresh"
       ++> hr

       ++> pageFlow "push" (push Append (disp tv) <** input tv)
       **> br
       ++> br
       ++> wlink () << b << "exit"

  where
  -- the widget being pushed:
  disp tv= do
      setTimeouts 100 0
      line ←  tget tv
      liftIO $ when (line ≡ "kill") $ myThreadId ↠ killThread
      p <<  line ++> noWidget

  -- The input box
  input tv= autoRefresh $ do
      line ←  getString Nothing <** submitButton "Enter"
      tput tv line


  tput tv x = atomic $ writeTVar  tv ( Just x)

  tget tv= atomic $ do
      mr ←  readTVar tv
      case mr of
         Nothing → retry
         Just r → do
          writeTVar tv Nothing
          return r

atomic= liftIO . atomically

There are still some minor issues with this widget, but I expect to fix them soon. As you can see, the push widget retry when Nothing is available in the TVar. When the input widget tput's something in the variable, it is read, emptied and returned to be displayed. The other widgets of the  page must be configured with autorefresh, unless we want to navigate away from the page. This is why the text input box and the button are under autoRefresh, but the exit link is not.

Automatic error trace generation in MFlow

To have  the trace of an unexpected error is very important in Web development. Specially when the error has been produced in exploitation. There is no way to make tests in a exploitation environment, so the error message is the only information available to fix it as soon as possible.

Now MFlow permits the creation of execution traces. Not just call traces, but execution traces, whenever an error happens. It uses the package monadloc from Pepe Iborra, used to produce stack traces in his package control-monad-exception. 

Using MonadLoc, MFlow can produce entire traces instead of call stacks because his backtracking mechanism permits to  run back the execution up to the beginning in case of an exception following the exact execution steps in reverse order. In this back-execution is when the trace is generated. When running normally, the tracing machinery does not affect the performance.

This is an example of what it is necessary in order to have execution traces in case of error.  It is necessary to install the monadloc-pp and monadloc packages that install the monadloc preprocessor and the monadloc class respectively. There are two according changes in the user programs, the preprocessor directive and to include  the Control.Monad.Loc module (in big letters):

This program has an intended error at line 33

1. {-# OPTIONS -F -pgmF MonadLoc #-}
2. module TestREST where
3. import MFlow.Wai.Blaze.Html.All
4. import Data.Monoid
5. import Data.String
6. import Control.Monad.Loc
7.  
8.  
9.  
10.  
11. main= runNavigation "" $ transientNav testREST
12.  
13.  
14. testREST= do
15.   setTimeouts 120 0
16.   liftIO $ print "start/restart"
17.  
18.   setHeader header1
19.  
20.   option ←  page $   wlink "a" << p << "letters " <++ p << "or"
21.                  <|> wlink "1" << p << "numbers"
22.  
23.   case option of
24.     "1" → do
25.           page $ wlink "2" << cont "1"
26.           page $ wlink "3" << cont "2"
27.           page $ wlink "4" << cont "3"
28.           page $ wlink ()  <<  "menu"
29.  
30.     "a" → do
31.           page $ wlink "b" << cont "a"
32.           page $ wlink "c" << cont "b"
33.           page $ undefined -- wlink "d" << cont "c"
34.           page $ wlink ()  <<  "menu"
35.  
36.  
37. cont x= p << "page for"
38.         <> b << x
39.         <> p << "goto next page"

If we navigate to execute this line with the web browser, the error produced in the console, in the file "errlog" (and in the Web browser if you are logged as administrator) is:

---------------------ERROR-------------------------
TIME=Tue Jul 16 11:52:16 Hora de verano romance 2013

TRACE (error in the last line):

testREST, TestREST(Demos\TestREST.hs): (14, 11)
testREST, TestREST(Demos\TestREST.hs): (20, 3)
testREST, TestREST(Demos\TestREST.hs): (23, 3)
testREST, TestREST(Demos\TestREST.hs): (30, 12)
testREST, TestREST(Demos\TestREST.hs): (31, 11)
testREST, TestREST(Demos\TestREST.hs): (32, 11)
testREST, TestREST(Demos\TestREST.hs): (33, 11)
exception: Prelude.undefined

USER= admin

VERB= navigation

REQUEST:
[("cookieuser","admin"),("flow","1373963274"),("Host","localhost"),("Connection","keep-alive"),("Cache-Control","max-age=0"),("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"),("User-Agent","Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36"),("Accept-Encoding","gzip,deflate,sdch"),("Accept-Language","es-ES,es;q=0.8,en-US;q=0.6"),("Cookie","flow=1373963274; cookieuser=admin")]

 

 

Fine, Isn´t? Note that it is not a call stack, but a true execution trace.
 

If we comment out both lines the error produced is:

 

---------------------ERROR-------------------------
TIME=Tue Jul 16 12:57:04 Hora de verano romance 2013

Prelude.undefined

USER= admin

VERB= navigation

REQUEST:

...

 

How I did that?

Essentially the monadLoc instance permits to add information to a monad about the line number that it is executing now.  I attach in this instance an exception handler to each bind operation so that when a noncaugh exception reaches the handler, it add this line number to  mfTrace, a list of  strings within the internal state monad. Then it trigger a failure that initiates the backtracking (see fail back monad) each step back detect the exception condition because mfTrace is not empty. then each of these steps add its own line to the mfTrace until the execution goes back to the init, were instead of restarting the execution forward, generate a error exception with all the content of mfTrace.

This is the instance:

instance  MonadLoc (FlowM v IO) where
    withLoc loc f = FlowM . BackT $ do
       withLoc loc $ do
            s ←  get
            (r,s') ←  lift $ do
                       (r,s') ←  runWithState f s  `CE.catch` (handler1  loc s)

                       case mfTrace s' of
                            Nothing     →  return (r,s')
                            Just  trace →  return(r, s'{mfTrace= Just( loc:trace)})
            put s'
            return r
       where

       handler1 loc s (e ∷  SomeException)=
        return (GoBack, s{mfTrace= Just ["exception: " ⊕show e]}) 

This is for the MFlow monad. For the View monad, used for in-page flows, the validation fails and mfTrace is filled with the error line, which trigger the backtracking in the MFlow monad.

The code of that last version with traces is not yet in Hackage. In github is at:

https://github.com/agocorona/MFlow

 

The fail-back monad is evolving to a supervisor monad that can execute tests, produce error traces , manage exceptional conditions, undo transactions and so on. I have to abstract out this monad from MFlow and make it available as a separate package.

In the next post I promise to talk about it.

I need people interested in MFlow to collaborate !!!!