Friday, November 10, 2017

Design Thinking vs. Traditional Analytics

Courtesy: Prof. Leidtka. 

iContract for Java...

  • Many programming languages have facilities to make assertions
  • iContract in Java is one such.
  • iContract is a preprocessor for Java. To use it, you first process your Java code with iContract, producing a set of decorated Java files. Then you compile the decorated Java code as usual with the Java compiler.
  • All iContract directives in Java code reside in class and method comments, just like Javadoc directives. In this way, iContract ensures complete backwards-compatibility with existing Java code, and you can always directly compile your Java code without the iContract assertions.
  • In a typical program lifecycle, you would move your system from a development environment into a test environment, then into a production environment. In the development environment, you would instrument your code with iContract assertions and run it. That way you can catch newly introduced bugs early on. In the test environment you may still want to keep the bulk of the assertions enabled, but you should take them out of performance-critical classes. Sometimes it even makes sense to keep some assertions enabled in a production environment, but only in classes that definitely are in no way critical to your system's performance. iContract allows you to explicitly select the classes that you want to instrument with assertions.
Pre and post conditions with iContract

Invariant with iContract


Agile Product Quality

Ensuring Product Quality in Agile

In his seminal book “Out of Crisis”, Edward Deming, the Quality Guru, summarized the Quality Management philosophy in 14 points. One of which is below:

Talking about inspection, he says “Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place”.

The bottom line is, inspections / reviews are an act too late into the product lifecycle, since they are done after the product reaches certain maturity. Therefore at best they can be a catalyst, but not by themselves assure quality. Not so different in this new world of Agile Development as well where inspections, reviews or another mechanism are all considered extrinsic to product development and so cannot completely assure product quality. Product quality it appears is built into the product.

Product development teams take charge of product quality assurance by adopting certain core engineering practices, thereby delivering Outcomes, not merely Outputs. The understanding being the quality of a product is intrinsic and is built into the product. Extending the logic, therefore, built-in quality practices will alone be able to improve product quality.

Some of the key aspects of Product Quality in the Agile context and how they are achieved are discussed below.

1. Requirements Traceability: Requirements traceability is intrinsic to Agile framework. There is a natural and strong hierarchy if we may call so, of user needs from Epic, Feature to User Story. And from there to DoD and further traceability to acceptance tests.

2. Architectural Runway: In Agile, architecture is done incrementally starting with an emergent design, which can be improved as sprints progress. This incremental approach to architecture instead of big-bang results in a model that is robust, yet is flexible to future changes.

3. Verification , Validation: Backlog grooming, Release Planning, Sprint planning, Daily stand ups, Product demos, and retrospective meetings Рall these key Agile ceremonies are done very frequently and iteratively in Agile and can be seen as Verification and Validation activities. Backlog grooming and sprint planning vigorously allow for debate on the needs, technical feasibility, prioritization, etc. The very fact that there is intense discussion around these topics and an AskàReviewàFollow-up loop ensures there is no ambiguity on the needs. Stand ups, retrospectives, and the level of communication among team members ensures the solutions, approach, plans etc. are continuously discussed & verified, and adapted. Product Demo ensures validation happens.

4. TDD and Refactoring: Only so much of code is written as to pass the written test case. This test-first approach guarantees the code has fewer bugs, is relatively clean, and most importantly, results in a code that is easy to maintain, flexible and can be easily extended

5. Pair programming: makes sure that there is live critique – reflect, correct, and progress on the code. This leads to increased code discipline, better code and collective code ownership, all of which benefits team cohesiveness.

6. Continuous Integration: Continuous integration ensures build and integration errors are detected early, easy to find and fix bugs, accelerates the build process, and results in full code visibility.

7. Continuous Delivery: Ensures enhancements, bug fixes, are rapidly and reliably pushed to production for a seamless experience to customers.

8. Definition of Done: Definition being binary – either achieved or not achieved, clearly removes ambiguity of what has and has not been achieved at Story, Sprint, Release levels.

9. Acceptance Criteria: Similar to DoD, Acceptance Criteria unambiguously gives the development team clarity on what it would take for the user story to be accepted as complete. Moreover, acceptance criteria can be easily translated to manual or automated test cases.

10. Continuous Improvement: Sprint on Sprint, Agile teams reflect and adapt their practices to suit and better their performance. Agile retrospectives in this sense, by design, allow and promote continuous improvement.

Product quality in Agile is a combined ownership, fostered by collaborative and proactive behaviors of team members.

Contract-First Design OR Design by Contract (DbC)

  • "DbC Is an approach for designing software. 
  • It prescribes that software designers should define formal, precise and verifiable interface specifications for software components, which extend the ordinary definition of abstract data types with preconditions, post conditions and invariants. These specs are referred to as Contracts…
  • The Design by Contract (DBC) software development technique ensures high-quality software by guaranteeing that every component of a system lives up to its expectations. 
  • Bertrand Meyer developed DBC as part of his Eiffel programming language."

Note: Double quoted text indicates the content is sourced from another site. Three essentials of DbC are:

1. Preconditions
2. Postconditions
3. Class invariants

  • Central to DBC is the notion of an assertion -- a Boolean expression about the state of a software system.
  • At runtime we evaluate the assertions at specific checkpoints during the system's execution. 
  • n a valid software system, all assertions evaluate to true. In other words, if any assertion evaluates to false, we consider the software system invalid or broken.
  • DBC's central notion somewhat relates to the #assert macro in C and C++ 

Extreme manufacturing (XM)

"The name was coined in 2012 by Joe Justice, founder of Wikispeed, and Marcin Jakubowski, founder of Open Source Ecology, as a take-off of the name extreme programming (XP), a software development methodology."

"eXtreme Manufacturing (XM) is an iterative and incremental framework for manufacturing improvement and new product development that was inspired by the software development methodology Scrum and the systematic waste-elimination (lean) production scheduling system Kanban"

The 10 core principles of XM are:

1. Optimize for change
2. OO, Modular Architecture
3. TDD
4. Contract-First Design
5. Design iteration
6. Agile h/w Design Patterns
7. CI Development
8. Continuously deployed development
9. Scaling patterns
10. Partner patterns

Sunday, October 29, 2017

The AI Podcast on SoundCloud...

The Agile Acronym Overload

As more and more frameworks get added they feel less like a valuable contribution and more like a way to make money.

When you go back to the core of Agile, it’s a simple set of four principles. You don’t need a replication of the London Underground tube map to understand it. (Referring to Deloitte's Agile Landscape)

Agile Framework is an oxymoron. If you adhere too strictly to any one framework, you’ve ceased to be Agile.

Agile is what you make it, not what people tell you it should be.

Agile doesn’t - and shouldn’t - work everywhere

-- Roisi Proven, Red Badger.

Tuesday, April 18, 2017

Microservice Architecture overheads

Courtesy: Benjamin Wootton
  1. Release and Deployment management of the set of Micro services requires a highly mature R&D automation support. 
  2. Operational overhead -- Testing, deploying and maintaining a set of microservices is time and energy intensive.
  3. For failover and resilience, Microservices need clustering, therefore a Monolith will now comprise tens of services (instead of 1 in a monolith). 
  4. Coupled with messaging layers, and loadbalancers, the landscape gets complicated. 
  5. Keeping this complex landscape functioning, by way of monitoring and production support would be daunting -- 1. Ensuring the processes stay up 2. Ensuring they don't consume up disk space + housekeeping 3. Deadlock management, etc. 
  6. All the above factors mean, a strong DevOps profile is needed, which may be hard to source. 
  7. Change to syntax or semantics of components results in a need to alter them on the other components as well. And to put these changes to production require a coordinated release. 
  8.   Suppose there is a need to recalculate tax differently for a particular product line. Now we have multiple options - - 1. Add a new service and call this service from other services. 2. Add the recalculation bit to all the services that require recalculation. 3. Have a tax calculating library (which requires parallel releases to ensure interface sanity) Either ways, the test and deployment effort is huge.
  9. This whole lot leads to a complex distributed system where network latency, fault tolerance, message serialisation, unreliable networks, asynchronicity, versioning, varying loads of application need to be taken care of. 
  10. Microservices architecture leads to asynchronous systems, however where synchronicity is desired,  managing correlation IDs / transaction IDs could be cumbersome. 
  11. Microservices landscape is dynamic with numerous inter-service interactions going on. Recreating test environments to carry out manual / automated tests may be difficult to achieve. 

Monday, April 17, 2017

Agile Progression

Microservices, SOA and APIs - IBM Article

Webservices and API

Enterprise Service Bus and Service Oriented Architecture


The ESB acronym and a related one - SOA - can be a source of confusion. ESB expands to an Enterprise Service Bus. SOA stands for Service Oriented Architecture.
That still doesn’t explain too much so here’s more information in plain English, without too much of corporate speak.

The whole truth

Think what happens when you log into your bank’s frontend application:
  1. Your name is displayed
  2. The account balance is there
  3. Your credit and debit cards are shown
  4. A list of your mutual funds can be there
  5. You get a pre-computed list of attractive loans you might be interested in
Now, it’s very likely that all these pieces belong to different systems and applications, each of which exposes data through an interface of some sort (HTTP, JSON, AMQP, XML, SOAP, FTP, CSV, doesn’t really matter):
  1. is from a CRM running on Linux and Oracle
  2. is from a COBOL system on z/OS mainframe
  3. is said to be from a mainframe but they’re too tight-lipped to tell you anything, only that they prefer CSV over anything else
  4. is from a mix of PHP and Ruby running on Windows
  5. is from PostgreSQL, Python and Java running on Linux and Solaris
The question now is, how do you make the frontend app talk to 1-5? Well, you don’t.
This is the fundamental basis of making sure such environments can scale above a handful of systems. You don’t let them talk to each other directly.
In the diagram below, each invocation of a service another system offers is represented in a line of different width or style:
Note that we didn’t even show you any higher-level processes (App1 invokes App2 and either App3 or App5 depending on whether an earlier response from App 6 was successful so that App4 can at a later time grab data produced by App 2 but only if App 1 doesn’t forbid it etc.).
Also note that we’re not talking about servers - each of the systems may run on 10 physical servers so there will be at least 60 physical components talking to each other.
Still, some questions become apparent.
How to separate interfaces? How can you plan rollouts? How do you coordinate updates or planned downtimes if each application is managed by different teams, vendors or departments and half of the original developers have left already?
If you think you can handle 6 applications, how about 30 of them?
Can you deal with 400? How about 2000? Each application can be a unique ecosystem requiring 10 servers or other devices to run on, so this is 20k moving parts spread over continents and all sort of technical or cultural boundaries, all that parts constantly and incessantly wishing to exchange messages and to chatter with each other all the time without any let-up, ever. (We’ll spare you a diagram)
There’s a good name for that situation. It’s called a mess.

How can you clean up the mess?

First thing is to honestly admit that the situation has gotten out of hand. This allows one to look for a remedy without feeling too much guilt. OK, it happened, you didn’t know any better, but there’s a chance it can be cleaned up.
This may mean an organizational change in an approach to IT but another step is to recall that systems and applications are not created merely to push data around. They’re meant to support business processes, regardless of what your business is, banking, audio recordings, radiolocation devices, anything.
Once you got these two clearly stated you can start thinking of building or redesigning your systems around services.
A service is something interesting, reusable and atomic that is offered by one system to other applications willing to make a good use of it, but it’s never exposed directly in a point-to-point manner. This is the shortest meaningful definition possible.
If a given functionality of a system fulfills these 3 requirements, that is, if it’s:
  • I nteresting
  • R eusable
  • A tomic
then there’s a very good chance it could and should be exposed as a service to other systems, though never directly.
Let’s discuss this IRA approach through a couple of examples.
EnvironmentAn electricity company’s CRM
FunctionalityReturning a list of customers that were active in a self-service portal in Q3 2012
Is it interesting?Yes, quite interesting. This can be used to generate all sorts of useful reports and statistics.
Is it reusable?No, not really. Although it does allow for creating higher-level constructs, such as statistics for a whole year, it’s clear that there won’t be much need for it in 2018.
Is it atomic?Most likely, yes. If there are similar services for other quarters, it will be possible to get an insight into the whole year
How to make it IRA?
  • Make it accept arbitrary start and stop dates instead of being limited to one quarter only
  • Make it accept arbitrary applications, not only the portal, let the application you’re interested in be an input parameter, it cannot be hard-coded to the portal only
Environmente-commerce site
FunctionalityReturning each piece of information ever collected regarding a given customer
Is it interesting?Well, yes. If you have access to the whole of it you’ll be always able to choose what you really need.
Is it reusable?Funnily enough, not exactly. There will be very few applications, if any, that will be interested in each and every bit of data.
Is it atomic?Definitely not. This monster functionality is bound to be logically composed of dozens of smaller parts.
How to make it IRA?
  • Split it into smaller pieces. Think what describes a customer - they have their addresses, phones, favorite products, contact methods they prefer and so on - each of these should be turned into an independent service.
  • Use ESB to create composite services out of the atomic ones
EnvironmentAny CRM anywhere
FunctionalityUpdating column CUST_AR_ZN in table C_NAZ_AJ after someone creates an account
Is it interesting?Absolutely not. This is a CRM’s internal function. No one in the sane world wants to deal with such low-level functionality.
Is it reusable?Yes, probably. An account can be created through multiple channels so this seems something reusable.
Is it atomic?Seems so, yes. It’s just a simple update of one column in one table.
How to make it IRA?Don’t even attempt to turn it into a service. It is not interesting. No one wants to think of particular columns and tables in one system. This is a CRM’s intricate detail, so even though it is reusable and atomic, you must not offer a service on top of it. It is your, the CRM’s, responsibility to think of it, don’t make anyone else shoulder it as well.
EnvironmentA mobile telco
FunctionalityRefilling a prepaid card in a billing system
Is it interesting?Extremely so. Everyone wants to use it, through text messages, IVR, IM, portals, gift cards etc.
Is it reusable?Very reusable, it can take part in all sort of higher-level processes
Is it atomic?Yes, from the calling application’s point of view, it can either refill a card or not. That a billing system will implement this functionality as a series of steps is irrelevant. From a business perspective it’s an atomic, indivisible service offered by a billing system.
How to make it IRA?It already is IRA.
If you’ve done any programming in the last 50 or so years it is now very clear that exposing a service is precisely like exposing an API in one part of code to another. The only difference is that you’re not dealing with submodules of a single system, you’re operating on a level of an entire environment of disparate systems.

Making services available on an ESB in an SOA

Now that you know that systems don’t exchange information directly and you understand what a service is, you can start making use of an ESB.
It now becomes the job of ESB to expose and invoke services of the integrated systems. That way, in most cases, only one access method, one interface, needs to be defined between each system and the ESB.
So if, like in the diagram above, you have 8 systems, there will be 16 interfaces (one in each direction) to create, maintain, manage and take care of.
Without an ESB you’d have 56 interfaces to think of and deal with (assuming each systems system talks with each other).
40 interfaces less means less time wasted and more money saved. That’s one of the reasons why your Fridays will be less strained.
This fact alone should make you strongly consider introducing an ESB.
If a system undergoes a rewrite, change of ownership, gets split between departments or vendors, it will be a duty of ESB folk to conform with the change. None of the other systems will even notice it because their interface with ESB will be left intact.
When you start breathe IRA services on a daily basis you can begin to think of composite ones.
Remember the ‘give-me-everything-you-can-about-that-customer’ kind of service above?
It wasn’t a good idea to create it but you will sometimes have to deal with client applications that need aggregated and summarized information. It will be ESB people who will be responsible for picking the best atomic services to build a composite one for that particular client system requiring this particular set of composite data.
With time the whole organization will begin to understand that it’s no more about database tables, files, batches, functions, routines or records. This will be about an architecture centered on interesting, reusable and atomic services applications offer to ESB.
No longer will people think that applications and system send things over one to another. They will see ESB as a universal access gate to interesting services their own systems can make use of. And they won’t even bother with checking who exactly provides what, their systems will only deal with ESB.
It takes time, patience and coordinated effort but it is doable.

But watch out for...

The best way to ruin the whole concept of SOA is to roll out an ESB and expect that matters will smooth out themselves. While still being a terrific idea, simply installing an ESB won’t achieve that much, unfortunately.
In the best case, sweeping something under the carpet, like in the diagram below, will achieve nothing.

Your IT people will loathe the system and the management will first tolerate an ESB as a new kid on the block but later on it will become a laughing stock. “What, that new silver bullet? Hahaha”.
Such consequences are inevitable if an ESB is not part of a bigger plan of actually moving things forward.

An ESB is for banks and similar only, then?

Not at all, no. It is a good choice in any situation requiring multiple data sources and multiple access methods to co-operate in order to achieve an interesting result.
For instance, grabbing latest readings from thermal sensors and publishing it into several channels, like e-mail alerts and an IPhone app does sound like a good fit for an integration platform.
Periodically consulting and monitoring whether all instances of a critical application are all up and if any is not, running a preconfigured script while sending a text message to admins also sounds just fine.
Everything that needs integration in a clear, well-defined environment is possibly a good fit for an ESB service but as always, deciding whether something really is a dream fit will follow from one’s experience. Naturally, Zato’s authors can help.

But I heard SOA was all about XML, SOAP and web services

Yes, this is what certain people would like you to believe.
If people or vendors you worked with did things such as BASE64-encoding a CSV file and sending it over to you in a SAML2-secured SOAP message then it’s quite understandable how you might’ve developed such an impression.
XML, SOAP and web services have their uses but like anything, they can be misused.
SOA is about a clean and manageable architecture. That a particular service might use SOAP or not is pretty much irrelevant. As an architectural approach, SOA will still be valid even if no SOAP service will be used at all.
If an architect designs a beautiful building, they can’t really do much about the color of the paint people choose for the interiors.
So no, SOA is not much about XML, SOAP and web services. These can be used too but it’s not the whole story behind it.
You are encouraged to kindly direct your strayed colleagues to this article to make them understand what SOA really is about.

And there’s more

This chapter covers only the very basics but should still give you a strong understanding how ESB and SOA should look like and what is needed to achieve a success.
Other topics, not covered here, include, but are not limited to:
  • How to get support from the management for an ESB
  • How to gather SOA architects and analytical teams
  • Introducing a Canonical Data Model (CDM) in an organization
  • Key Performance Indicators (KPI) - now that you have a common and unified method of providing services across systems, you should start to monitor and evaluate what is really delivered to you
  • Business Process Management (BPM) - how and when to choose a BPM platform to orchestrate the services (answer - not too soon, get yourself familiar with how to build nice and adorable services first)
  • What to do with systems that have no APIs? E.g. should an ESB access their databases directly (answer - it depends, there’s no golden rule)

Saturday, April 15, 2017

Agile Scaling

Unless the Top and Bottom layers are fixed, simply developing software in an iterative-incremental framework means nothing. There is, therefore, a need for complete organizational transformation spanning all the three layers. 

Bitcoin basics

Blockchain Technology


Cloud Microservices - IBM

Serverless Architecture

BaaS and FaaS

Courtesy:, and Youtube

  1. Serverless refers to applications that significantly or fully depend on 3rd party applications / services (‘in the cloud’) to manage server-side logic and state. These are typically ‘rich client’ applications (think single page web apps, or mobile apps) that use the vast ecosystem of cloud accessible databases (like Parse, Firebase), authentication services (Auth0, AWS Cognito), etc. These types of services have been previously described as ‘(Mobile) Backend as a Service’.
  2. Serverless can also mean applications where some amount of server-side logic is still written by the application developer but unlike traditional architectures is run in stateless compute containers that are event-triggered, ephemeral (may only last for one invocation), and fully managed by a 3rd party. One way to think of this is ‘Functions as a service / FaaS’ . AWS Lambda is one of the most popular implementations of FaaS at present, but there are others. I’ll be using ‘FaaS’ as a shorthand for this meaning of Serverless throughout the rest of this article.