Archive | Big Data RSS feed for this section

Six Areas Where AI Is Improving Customer Experiences

30 Apr

Bottom Line: This year’s hard reset is amplifying how vital customer relationships are and how much potential AI has to find new ways to improve them.

  • 30% of customers will leave a brand and never come back because of a bad experience.
  • 27% of companies say improving their customer intelligence and data efforts are their highest priority when it comes to customer experience (CX).
  • By 2023, 30% of customer service organizations will deliver proactive customer services by using AI-enabled process orchestration and continuous intelligence, according to Gartner.
  • $13.9B was invested in CX-focused AI and $42.7B in CX-focused Big Data and analytics in 2019, with both expected to grow to $90B in 2022, according to IDC.

The hard reset every company is going through today is making senior management teams re-evaluate every line item and expense, especially in marketing. Spending on Customer Experience is getting re-evaluated as are supporting AI, analytics, business intelligence (BI), and machine learning projects and spending. Marketers able to quantify their contributions to revenue gains are succeeding the most at defending their budgets.

Fundamentals of CX Economics

Knowing if and by how much CX initiatives and strategies are paying off has been elusive. Fortunately, there are a variety of benchmarks and supporting methodologies being developed that contextualize the contribution of CX. KPMG’s recent study, How Much Is Customer Experience Worth? provides guidance in the areas of CX and its supporting economics. The following table provides an overview of key financial measures’ interrelationships with CX. The table below summarizes their findings:

The KPMG study also found that failing to meet customer expectations is two times more destructive than exceeding them. That’s a powerful argument for having AI and machine learning ingrained into CX company-wide. The following graphic quantifies the economic value of improving CX:

Where AI Is Improving CX

For AI projects to make it through the budgeting crucible that the COVID-19 pandemic has created, they’re going to have to show a contribution to revenue, cost reduction, and improved customer experiences in a contactless world. Add in the need for any CX strategy to be on a resilient, proven platform and the future of marketing comes into focus. Examples of platforms and customer-centric digital transformation networks that can help re-center an organization on data- and AI-driven customer insights include BMC’s Autonomous Digital Enterprise (ADE) and others. The framework is differentiated from many others in how it is designed to capitalize on AI and Machine Learning’s core strengths to improve every aspect of the customer (CX) and  employee experience (EX). BMC believes that providing employees with the digital resources they need to excel at their jobs also delivers excellent customer experiences.

Having worked my way through college in customer service roles, I can attest to how valuable having the right digital resources are for serving customers What I like about their framework is how they’re trying to go beyond just satisfying customers, they’re wanting to delight them. BMC calls this delivering a transcendent customer experience. From my collegiate career doing customer service, I recall the e-mails delighted customers sent to my bosses that would be posted along a wall in our offices. In customer service and customer experience, you get what you give. Having customer service reps like my younger self on the front line able to get resources and support they need to deliver more authentic and responsive support is key. I see BMC’s ADE doing the same by ensuring a scalable CX strategy that retains its authenticity even as response times shrink and customer volume increases.

The following are six ways AI can improve customer experiences:

  • Improving contactless personalized customer care is considered one of the most valuable areas where AI is improving customer experiences. These “need to do” marketing areas have the highest complexity and highest benefit. Marketers haven’t been putting as much emphasis on the “must do” areas of high benefit and low complexity, according to Capgemini’s analysis. These application areas include Chatbots and virtual assistants, reducing revenue churn, facial recognition and product and services recommendations. Source:  Turning AI into concrete value: the successful implementers’ toolkit, Capgemini Consulting. (PDF, 28 pp).
  • Anticipating and predicting how each customers’ preferences of where, when, and what they will buy will change and removing roadblocks well ahead of time for them. Reducing the friction customers face when they’re attempting to buy within a channel they’ve never purchased through before can’t be left to chance. Using augmented, predictive analytics to generate insights in real-time to customize the marketing mix for every individual Customer improves sales funnels, preserves margins, and can increase sales velocity.
  • Knowing which customer touchpoints are the most and least effective in improving CX and driving repurchase rates. Successfully using AI to improve CX needs to be based on data from all trackable channels that prospects and customers interact with. Digital touchpoints, including mobile app usage, social media, and website visits, all need to be aggregated into data sets ML algorithms to use to learn more about every Customer continually and anticipate which touchpoint is the most valuable to them and why. Knowing how touchpoints stack up from a customer’s point of view immediately says which channels are doing well and which need improvement.
  • Recruiting new customer segments by using CX improvements to gain them as prospects and then convert them to customers. AI and ML have been used for customer segmentation for years. Online retailers are using AI to identify which CX enhancements on their mobile apps, websites, and customer care systems are the most likely to attract new customers.
  • Retailers are combining personalization, AI-based pattern matching, and product-based recommendation engines in their mobile apps enabling shoppers to try on garments they’re interested in buying virtually. Machine learning excels at pattern recognition, and AI is well-suited for fine-tuning recommendation engines, which are together leading to a new generation of shopping apps where customers can virtually try on any garment. The app learns what shoppers most prefer and also evaluates image quality in real-time, and then recommends either purchase online or in a store. Source: Capgemini, Building The Retail Superstar: How unleashing AI across functions offers a multi-billion dollar opportunity.
  • Relying on AI to best understand customers and redefine IT and Operations Management infrastructure to support them is a true test of how customer-centric a business is. Digital transformation networks need to support every touchpoint of the customer experience. They must have AI and ML designed to anticipate customer needs and deliver the goods and services required at the right time, via the Customer’s preferred channel. BMC’s Autonomous Digital Enterprise Framework is a case in point. Source: Cognizant, The 2020 Customer Experience.

Japan: world leaders in robots and growing old

8 Nov

Japan is a global leader in two opposing growth dynamics – a declining workforce, which is a drag, and robotics, which is beneficial to productivity.


The adverse implications of an ageing and declining population for growth are behind the Abe administration’s ambitious Society 5.0 strategy.

” In 2015, 27 per cent of Japan’s population was older than 65. This is expected to rise by over 10 percentage points to 38 per cent by 2050.”

The adverse implications of an ageing and declining population for growth are behind the Abe administration’s ambitious Society 5.0 strategy.

Society 5.0 envisages greater adoption of artificial intelligence (AI), robotics and big data to enhance long-term productivity. These technologies will help fill the void of a declining workforce and/or augment the existing labour force.

With a number of other leading economies also facing declining and ageing populations, Japan’s endeavour should provide a useful case study for the future of work.


Below 100 million

Japan’s population fell by a record-breaking 264,000 in 2018 to 126.4 million. With little prospect of immigration being on the government’s agenda, it will decline further in coming years given the current birth rate (1.4) is significantly below the steady-state rate (2.1). By 2050 Japan’s population is expected to fall below 100 million.

In 2015, 27 per cent of Japan’s population was older than 65. On current trends this will rise by over 10 percentage points to 38 per cent by 2050.

But there will be more robots. Automation and robotics are not new for Japan. The nation has long been a world leader in technological development, especially in robot technology. In 2018, Japan exported $US2 billion worth of industrial robots.

This was more than the next five largest exporters (Germany, Italy, France, China, Denmark) combined.

It is one of the most robot-integrated economies in the world in terms of “robot density” – measured as the number of robots relative to humans in manufacturing and industry.


Lack of pressure

Traditionally much of Japan’s investment in robotics has been in the export orientated manufacturing sectors, especially automotive and electronics, where automation features significantly in the production process. Very little investment has been made by the much larger services sector, which accounts for almost three quarters of the economy.

Indeed the lack of technology investment by the service sector likely contributes to the sizeable productivity gap between the manufacturing and service sectors.

There has been very little productivity growth in the services sector over the past couple of decades. This lack of investment may partly reflect the fragmented state of many service-based industries and the lack of competitive domestic pressure.


Productivity in the services sector has also lagged that of other main advanced economies. Notably the labour productivity of the non-manufacturing sector in Japan is about 60 per cent of that of the United States.

There would seem room for both a catch-up and an organic improvement in the underlying productivity of the service sector. This should yield substantial dividends to gross domestic product (GDP) growth, given the service sector accounts for nearly three quarters of the economy.

Future of work

Japan’s Society 5.0 envisages a super-smart society where technologies such as big data, Internet of Things (IoT), AI and robots are present in every industry and across all social segments. This revolution would make everyday life more comfortable, efficient and sustainable.

As part of this integrated strategy, the government has produced a number of detailed and ambitious reports, including: IT Strategy for Data UtilizationRobot Strategy and an Artificial Intelligence Technology Strategy.

Recent surveys highlight a pick-up in both actual and planned capital expenditure on new technology. This trend is notable for small and medium enterprises that need to compensate for scarce labour while staying competitive.

Game-changing five

Japan’s comprehensive blueprint for Society 5.0 includes strategic objectives, implementation scenarios and key performance indicators. Some examples of current and future integration include:

• electronic payments and self-checkout registers in retail outlets;

• touch-screen menus in hospitality to streamline operations;

• drones to deliver goods in remote areas, survey property and support disaster relief;

• online medical care to enhance best practice, reduce travel, increase support to less-mobile patients and conveniently offer 24/7 monitoring, including nursing robots; and

• autonomous transport (driverless buses, cars and trains).

Robotic invasion?

The empirical evidence on the impact of automation and technology on jobs is mixed. In the short term, some workers are more vulnerable to displacement, so there are likely to be transition costs leading to undesirable consequences of lost income, income polarisation and rising inequality.

In the long run, however, technological advances boost productivity, which over time creates new jobs, allowing incomes and living standards to rise.

A 2017 RIETI discussion paper, using Japanese prefectural data, found increased robot density in manufacturing to be associated not only with greater productivity but also with local gains in employment and wages. This suggests embracing innovation outside of manufacturing should also provide long-term dividends. Technical innovation is also necessary to help alleviate a declining workforce.

That said, the Japanese government will need to carefully manage the transition.

Strong and effective social safety nets will be crucial to support workers displaced or disadvantaged. In addition, the government can take a proactive position in educating and reskilling workers to enable them to take advantage of jobs in a high-tech world.

Increasing technological change in Japan will affect a spectrum of industries and improve quality of living. Japan is a relatively unique case in the world given its negative labour-force dynamics.

Productivity supported by investment in automation, AI and technology will need to feature strongly as an engine supporting long-term economic growth. Japan’s experience could hold valuable lessons for economies such as China, South Korea and Europe, which are facing similar demographic trends.

08 11 19

11 Big Data Trends for 2020: Current Predictions You Should Know

9 Oct

Big data and analytics (BDA) is a crucial resource for public and private enterprises nowadays. Thanks in large part to the evolution of cloud software, organizations can now track and analyze volumes of business data in real-time and make the necessary adjustments to their business processes accordingly. As the industry goes deeper into the age of AI, what big data trends should businesses be most wary of?

Given that the BDA market is projected to become a more lucrative field in the following years, what does this mean to the way you should be conducting business moving forward? Should you be looking into harnessing BDA to move your business forward? Here are eleven big data trends impacting the current landscape to help you see the bigger picture.

big data trends web main

Further along, various businesses will save $1 trillion through IoT by 2020 alone. Businesses with hypercomplex processes, multiple branches, departments and thousands of teams will benefit the most when smart structures, machines and gadgets do most of the necessary adjustments by themselves.

Data Analytics Top 4 Benefits 2019

Faster innovation cycles 25%

Improved business efficiencies 17%

More effective R&D 13%

Product/service 12%

Source: Chicago Analytics Group

However, the figures for losses are more pronounced than those for the winners.

For example, poor data quality alone will cost the US economy $3.1 trillion a year.

That’s already more than the GDP of many countries, but it’s further confounded by 91% of companies who feel they are consistently wasting revenue because of their poor data.

This is no longer a normal global economy we are witnessing in our lifetime. Ecommerce and online carts have already obliterated thousands if not millions of businesses big and small all over the world. Watch out for how many of them will further fall by the wayside because of poor understanding of all the data they have.

1. Riding the wave of digital transformation

Digital transformation is the global currency pushing technology all over the world. That much work done and work still to do leaves a trail of data the volume of which is pretty much unheard of in human history.

It will continue to grow as IaaS providers scamper to cover the ground and build data centers. They will do so from the bowels of the ocean or to the literal ends of the earth–the polar regions–to drive away heat which is data centers’ constant challenge.

Digital transformation goes hand in hand with the Internet of Things (IoT), artificial intelligence (AI), machine learning and big data.

With IoT connected devices expected to reach a staggering 75 billion devices in 2025 from 26.7 billion currently, it’s easy to see where that big data is coming from.

Machine learning and AI tools will try to rein in that much big data spewing out of the massive data centers from operating the systems, making sense of the hidden relationships, and storing and projecting the insights within the bounds of human understanding.

Still, corporations have much work to do optimizing the use of all that data on their data servers. In the US economy alone, for example, they are losing as much as $3.1 trillion a year from the cost of poor data quality. It remains to be seen how these enterprises are going to address that.

Key takeaways

  • Digital transformation in the form of IoT, IaaS, AI and machine learning is feeding big data and pushing it to territories unheard of in human history;
  • IoT connected devices alone will reach a point where there would be multiple connected devices within homes and buildings for each person who will ever live;
  • Humans still have much to learn to make sense out of all that data. AI and machine learning—and the looming arrival of quantum computers—are seen as the best bets to accomplish all that.

2. Big data to help climate change research

Backing up the views and predictions of climate change organizations like the UN Intergovernmental Climate Change (IPCC) with solid data will put the raging climate change debate to rest. In the aftermath, nations will finally work together to execute the requisite actions needed to save the planet.

That is not to say that the data might also show other interesting insights about what’s really going on with the planet’s climate. Whatever the case, none of it will be legitimate without the presence of cold data exempt from the biases of humans hailing from either side of the climate change debate.

Humans would like to know whether carbon dioxide emissions are all there is to know about climate change. Who knows whether looking at the faraway galaxies might reveal some patterns about the solar system’s path along with the Milky Way’s regular celestial rotation?

We would like to know and that entails unimaginable data input from all the giant scientific observatories stationed on earth and its atmosphere.

Not only that: we would also have to incorporate unimaginably massive inputs from ocean research, earth sciences, meteorological research centers, and perhaps even from the mind-boggling nuclear research facilities as they approximate events from the Big Bang to the current age of the universe.

The stake of businesses

Why should businesses worry about climate change?

For one, agriculture production would be most affected by even the tiniest drop in the local temperature.

Two, severe climate change will drastically impact the health of populations worldwide. What that means for businesses everywhere is much too deep to even contemplate.

Drained resources? Check. Massive population movement? Check. Massive lands submerged in oceans? Check. Food security thrown out the window? Check. Governments unable to meet the devastating changes to their lands and populations? Check.

In the face of all that, where would businesses go?

No matter which side of the climate change debate you happen to be with, a few thoughts stand out:

Key takeaways

  • Big data is crucial to the climate change debate, especially data with no set bias to begin with;
  • Big data to establish the climate change truth will come from disparate research facilities all over the world, ranging from the earth sciences, particle physics research centers to ocean research data sets;
  • There is much at stake for businesses in the climate change debate.

3. Real-time analytics gains more traction

data-heavy streaming gains further traction

Tennis and other major global sports show the tremendous capability of data-heavy live data analytics streaming.

Apart from the scintillating game served up by the Djokovich-Federer match during the 2019 Wimbledon final, the viewers were also thrilled by the constant feed of live statistics immediately related to the on-court drama transpiring before their eyes. Those who were casual followers of the game were caught up in the clash of numbers that described the unfolding play. For a fleeting moment, they became expert analysts without the extraneous ad-libs so commonly dished out by live commentators.

For the sections of the audience who were rooting for Federer, they all but won everything except the trophy. Federer was ahead in the stats that matter, except the clutch plays that matter most when the trophy was on the line. So Djokovic took the trophy and left thousands if not millions of Federer fans watching in tears. Interesting, nerve-racking watch.

But the live statistics presentations may be more interesting for a number of reasons.

More than just tennis

For one, they go beyond tennis or any other sport that uses them—the NBA and football have been using them too, as do other major sports.

Beyond sports, think what the financial world could do with such immense power—to comb through petabytes of live data coursing through intricate network connections and finally to the servers that work with countless other devices to produce the tantalizing numerical reports. See an ongoing financial fraud as they are committed in concert by linked criminals all over the world? Check.

How about helping with earthquakes and other natural disaster prediction and prevention? Big data, AI and machine learning are working together to finally solve this natural world riddle.

In the meantime, organizations like Oracle are leveraging robotic process automation (RPA), machine learning and visual big data analysis to thwart increasingly sophisticated criminal activities in the financial sector.

Addressing El Niño

El Niño and other tremendous weather anomalies next get the AI and big data treatment. The latest development on the field is grabbing the headlines, with predictive capability going as deep as 18 months in advance.

Key takeaways

  • Big data is already well in position to become a regular sports feature in presenting data-heavy streaming data analytics to audiences.
  • Organizations that oversee critical research on earthquakes, El Niño and other natural phenomena will increasingly rely on big data with the help of AI, RPA and machine learning to come out with extremely useful predictions.
  • The financial sector is one of the industries to immediately benefit from this big data trend.

4. Big Data is heading to stores near you

No, not really, but it’s a great metaphor for how data-as-a-service is becoming almost as commonplace as the proverbial mom-and-pop stores that once covered the entire landscape of the USA. How commonplace? In the region of 90% of enterprises getting into the action and generating revenue from it.

Data-as-a-service (DaaS) is really nothing new or revolutionary—you’ve probably encountered it in the form of purchased music, videos, or image files from multiple sources online—but the entry of a whole lot of new players from map data providers to product catalog vendors changes the whole concept completely.

It doesn’t have to be just dedicated SaaS software solutions getting on the act too: if you have a company whose data could mean something to others—okay, hello Cambridge Analytica— or have a hard time maintaining it, your best bet is selling it per megabyte, per specific file format, or by volume quotes.

Since data resides in the cloud, you could well be atop Timbuktu and have a play of the latest Netflix show when the clouds are not too kind to give you a spotless view of the stars.

Key takeaways

  • Simplified access – customers can access the data using any device and from anywhere in the world
  • Cost-effective – You can simply outsource your data to other companies who will build the presentation interface at a minimal cost.
  • Easy update – By keeping data in a secured, single location, it’s easy to update any one of them quickly and conveniently.

5. Usher businesses to new areas of growth

Analytics in the form of business intelligence solutions has been helping businesses for a time now. While the numbers have been impressive thus far, the new generation of this software should allow new and old customers to scale new heights.

Data Analytics Top 4 Benefits 2019

Faster innovation cycles 25%

Improved business efficiencies 17%

More effective R&D 13%

Product/service 12%

Source: Chicago Analytics Group

The new trend in integrating every critical aspect of business operation from advertising, supply chain management, support and social media management among others.

The vast amount of data involved could be from landing page behavior patterns, customer transactions, geographical origins, video feeds from multiple store branches, customer survey results and the like. No matter, the new analytic tools should plow through them even in real time and produce insights that are not possible with many offerings today.

While Netflix grabs the headline among the early winners of big data analytics adoption, the future will expand the list of those making the most of taking the numbers game to the highest levels.

Retailers already realize increased margins of up to 60% with current analytics methodologies. The addition of the aforementioned capabilities in tandem with location-aware and located-based services should see the numbers shoot up even more.

Key takeaways

  • A new generation of analytic tools should help businesses scale new revenues levels;
  • The new generation of business analytic tools would take a holistic approach to all business processes;
  • Location-aware tools would spearhead this new analytic development.

6. Big data to search for novel medical cures

Businesses have much interest in investing in human welfare. Healthy populations allow them to hire healthy workers and lessen the burden on health-induced absences, payments and other work-related issues.

An alarming piece of data is that in the US alone, healthcare expenses now account for 17.6 percent of its GDP. It thus makes sense that one of the raging applications of big data is on the field of medicine. With the number of human maladies old and new popping up around the world, the role of big data in this industry is only to grow further.

Many scientists hope that by consolidating all the medical records ever accumulated on the planet, the speed of finding medical cures will become faster and sooner than expected. The challenge is to find a middle ground among research institutions private and public throwing patents all over the place and slowing down the process of finding new discoveries.

Consolidating all medical data is easier said than done, too. Data containing clinical records go in the vicinity of 170 exabytes for 2019 alone, with yearly increase of about 1.2 to 2.4 exabytes per year. Getting around all that vast zeroes and ones is no mean feat but the rewards are more than worth it.

Early successes

This early there are promising studies in various research laboratories to cure cancer and aging, with Silicon Valley stalwarts actively getting on in the last part. Variously called immortality project or longevity research, vast amounts of money and brain talent are being thrown to make this vision come true within their lifetimes.

Vast libraries of DNA records, patient records, research studies, and other related fields are accessed to get AI to make connections and perhaps come out with new medications altogether.

More: big data is fueling research on improving staffing of medical facilities, storing and automatically processing access to mountains of electronic health records and allowing for real-time alerts of patient status.

As for cancer itself, big data has already produced an unexpected finding, discovering that the anti-depressant Desipramine is capable of healing certain types of lung cancer, for example.

Key takeaways

  • Big data is perceived as the key to unlocking the long-sought cures to human diseases, cancer among them.
  • Silicon Valley big names are actively contributing to the intense research especially in human longevity research.
  • Probes into medical big data are already producing unexpected positive results.

7. Big data cuts travel time

Admittedly, full autonomous driving is still a long way from truly taking off. However, processing big data fed by call data records (CDRs) from mobile data users to optimize travel routes and estimates could be the next best thing. This is especially applicable to the worst traffic-hit cities in the world.

With the right analytic tools, the enormous traffic big data could shed light on trip generation and commuter transportation management. Tracking the locations and matching the origins and target destinations should give travelers the opportunity to calculate their travel times better.

The powerful algorithms should have no trouble crunching the numbers. This could be to monitor city traffic in real time and identify congested routes and recommend alternative roads instead.

The cost of congestion is appalling. In 2017 alone, the United States, the UK and Germany lost $461 billion due to traffic. That figure is equivalent to $975 per person.

Source: TomTom International BV

The number could balloon to $2.8 trillion for the US come 2030. This places technology at the heart of the solution, along with better urban planning and traffic management.

8. Simulate oil fields or the quantum realm

One of the biggest beneficiaries of big data analytics is the petroleum industry. With exascale computing power now within reach of oil companies, they have a better tool to probe into the enormous amount of data generated by seismic sensors.

Meanwhile, high-fidelity imaging technologies and new algorithms to simulate models give them an unprecedented level of clarity into the potential of reservoirs under exploration. With clearer information on hand, they minimize risks identifying and mapping oil reservoirs and optimizing management and operational costs.

In one such case, a large oil and gas company reduced operational costs by 37% after the introduction of big data analytics.

Into the quantum realm

The same advances in processing, I/O solutions and networking allow us to model spatial scales from the subatomic realm to the supergalactic clusters. We can even add at the scale of the universe or multiverse if it comes to that.

In terms of timescales, the combination of big data, machine learning and AI is opening up portals to the scales of femtoseconds to eons.

While deep research into these quantum realms does not give businesses immediate windfalls, they will most likely play a big part in the activities now reaching frenetic proportions. We are talking about corporations and nations already casting their eyes on future space mining ventures.

Key takeaways

  • Petroleum industries are saving themselves from risk exposure and high operational costs through big data analytics;
  • The use of simulation will impact other businesses with the arrival of cutting-edge technologies. These include advanced algorithms, faster networking, new I/O solutions among others.
  • The potential of space mining is nudging countries and businesses to be the first to establish unprecedented space mining investments.

9. More natural language processing

Big data, AI, IoT, machine learning are pushing the boundaries of human and technological interaction. It gives these technologies a human face through natural language processing (NLP).

While populations have become enamored with technologies in general, there is a pervading sense of a line clearly drawn between gadgets and humans. Technophobes will perhaps not get their David-class Osment’s flavor of AI to love soon. However, natural processing should give this class of technology a warmer face and further adoption than their more dystopian Blade Runner versions.

And at their current state, natural processing is not going android or cyborg soon. Instead, they will help people engage and interact with various smart systems with nothing but human language. The more advanced of them will do so with a level that comes with the nuances of the language in use.

NLP will allow even the most casual users to interact with intelligent systems. They don’t have to resort to exotic codes which is the typical way it is done. Not only access to quality information, too. They can also prompt the system to give them the insights they need to move forward.  The content will be delivered in human voice if they so choose it. They can also opt for the summaries to be read to them even while they are on the go.

NLP can give businesses access to sentiment analysis. It will allow them to know how their customers feel about their brands at a much deeper level. There are many ways the information can then be tied to specific demographics, income levels, educational demographics and the like.

Augmented data management

In the same vein, augmented data management will also see a rise in importance within companies. This will happen as AI becomes more efficient with enterprise information management categories. These include data quality, metadata management and master data management among others. This means that manual data management tasks will be lessened. All of it thanks to ML and AI developments, enabling specialists to take care of more high-value tasks.

That said, companies looking to utilize this innovative technology should carefully review the available augmented data management and data analytics tools in the market that best fits their business operations. This way, they can properly integrate such solutions into their business processes and properly harness the big data.

investment in augmented analytics

Key takeaways

  • NLP will give casual users access to crucial information previously inaccessible to them. This without learning esoteric machine language to interact with the computer systems;
  • NLP will allow businesses to process customer sentiment. This is a very powerful tool to identify the needs of clients and design products and services around them;
  • Augmented analytics will allow decision-makers to focus on business matters that truly matter.

10. Data governance moves forward

Following the introduction of the General Data Protection Regulation (GDPR) guidelines last year, data governance initiatives continue to mobilize globally. This means more uniform compliance for all business sectors that handle big data. Otherwise, they face a substantial fine and other penalties.

This compliance comes after recent 2018 studies show that 70% of surveyed businesses worldwide failed to address requests by individuals who want to get a copy of their personal data as required by GDPR within the one-month time limit set out in the regulations.

When companies are more forthright handling customer data while limiting what they can do with it, people will be encouraged to trust online payment transactions than ever before.

Power in the hands of customers

GDPR places the power back in the hands of customers. This is done by appointing them as the firm owners of any information they create. It gives them the right to cart away their data from a misbehaving business. They can then give it to another who appreciates doing clean business with them better.

Moreover, companies and businesses shouldn’t just worry about getting fined if they fail to comply with GDPR regulations.

The effects of GDPR is a two-way street. Companies that comply will see positive effects on their brand reputations. This is most likely as customers vote trustworthy vendors with their wallets.

Trustworthy businesses will generate more reliable big data. This ensures that any analytics thrust into the data sets will come out with solid bases.

Key takeaways

  • GDPR empowers consumers while protecting their right to their own data;
  • Businesses that are more forthright handling customer data will be amply rewarded in the markets;
  • GDPR makes big data cleaner and capable of producing more dependable analysis results.

11. Cybersecurity remains a challenge

When you pair big data with security, it’s too easy to fall for popular clichés. Among these is: “The bigger they are, the harder they fall.”  How about “With great power comes great responsibility”?

And yet the events at Yahoo wherein 3 billion accounts were compromised and the much-publicized Facebook and Cambridge Analytica fiasco remind us that when it comes to our private data, nothing is ever small and safe at the same time.

top data breaches

In this day and age where the world pays dearly for not properly addressing cybersecurity flaws to the tune of $2 trillion, it’s much easy to become paranoid with sending financial codes over the internet superstructure.

Businesses and organizations have many cybersecurity challenges in their hands. Most likely it’s one aspect of big data that will linger longer than we would like to hear about.

Non-relational databases, limited storage options, distributed frameworks are just some of the most lingering challenges of big data.

With big data becoming more and more of a lucrative resource, it is prudent that companies of all sizes should look into and invest in reliable cybersecurity software providers in order to protect such valuable business information from cyberattacks.

Key takeaways

  • Cybersecuritychallenge will grow in number and complexity as the volume of data that it targets;
  • Cybercriminals have a number of options to attack big data from multiple processes and vantage points.
  • Cybersecurity and cybercriminals are playing an unending cat-and-mouse chase game.

Use Big Data or Perish

As we are now more than halfway into 2019, we can expect further developments in big data analytics. Much of data use will be regulated and monitored in both the private and public sectors.

Based on the market projections, big data will continue to grow. This will affect the way companies and organizations look at business information. Companies should be keen on bolstering their efforts to adapt their business operations. For that, they can begin to optimize the use of information with analytical software. The objective is to make their businesses grow while transforming their data-driven environment. As such, it is best to keep up-to-date with the latest big data research and news.

Using R for Scalable Data Analytics

1 Apr

At the recent Strata conference in San Jose, several members of the Microsoft Data Science team presented the tutorial Using R for Scalable Data Analytics: Single Machines to Spark Clusters. The materials are all available online, including the presentation slides and hands-on R scripts. You can follow along with the materials at home, using the Data Science Virtual Machine for Linux, which provides all the necessary components like Spark and Microsoft R Server. (If you don’t already have an Azure account, you can get $200 credit with the Azure free trial.)

The tutorial covers many different techniques for training predictive models at scale, and deploying the trained models as predictive engines within production environments. Among the technologies you’ll use are Microsoft R Server running on Spark, the SparkR package, the sparklyr package and H20 (via the rsparkling package). It also touches on some non-Spark methods, like the bigmemory and ff packages for R (and various other packages that make use of them), and using the foreach package for coarse-grained parallel computations. You’ll also learn how to create prediction engines from these trained models using the mrsdeploy package.


The tutorial also includes scripts for comparing the performance of these various techniques, both for training the predictive model:


and for generating predictions from the trained model:


(The above tests used 4 worker nodes and 1 edge node, all with with 16 cores and 112Gb of RAM.)

You can find the tutorial details, including slides and scripts, at the link below.

Strata + Hadoop World 2017, San Jose: Using R for scalable data analytics: From single machines to Hadoop Spark clusters



Streaming Big Data: Storm, Spark and Samza

1 Apr

There are a number of distributed computation systems that can process Big Data in real time or near-real time. This article will start with a short description of three Apache frameworks, and attempt to provide a quick, high-level overview of some of their similarities and differences.

Apache Storm

In Storm, you design a graph of real-time computation called a topology, and feed it to the cluster where the master node will distribute the code among worker nodes to execute it. In a topology, data is passed around between spouts that emit data streams as immutable sets of key-value pairs called tuples, and bolts that transform those streams (count, filter etc.). Bolts themselves can optionally emit data to other bolts down the processing pipeline.


Apache Spark

Spark Streaming (an extension of the core Spark API) doesn’t process streams one at a time like Storm. Instead, it slices them in small batches of time intervals before processing them. The Spark abstraction for a continuous stream of data is called a DStream (for Discretized Stream). A DStream is a micro-batch of RDDs (Resilient Distributed Datasets). RDDs are distributed collections that can be operated in parallel by arbitrary functions and by transformations over a sliding window of data (windowed computations).


Apache Samza

Samza ’s approach to streaming is to process messages as they are received, one at a time. Samza’s stream primitive is not a tuple or a Dstream, but a message. Streams are divided into partitions and each partition is an ordered sequence of read-only messages with each message having a unique ID (offset). The system also supports batching, i.e. consuming several messages from the same stream partition in sequence. Samza`s Execution & Streaming modules are both pluggable, although Samza typically relies on Hadoop’s YARN (Yet Another Resource Negotiator) and Apache Kafka.


Common Ground

All three real-time computation systems are open-source, low-latencydistributed, scalable and fault-tolerant. They all allow you to run your stream processing code through parallel tasks distributed across a cluster of computing machines with fail-over capabilities. They also provide simple APIs to abstract the complexity of the underlying implementations.

The three frameworks use different vocabularies for similar concepts:


Comparison Matrix

A few of the differences are summarized in the table below:


There are three general categories of delivery patterns:

  1. At-most-once: messages may be lost. This is usually the least desirable outcome.
  2. At-least-once: messages may be redelivered (no loss, but duplicates). This is good enough for many use cases.
  3. Exactly-once: each message is delivered once and only once (no loss, no duplicates). This is a desirable feature although difficult to guarantee in all cases.

Another aspect is state management. There are different strategies to store state. Spark Streaming writes data into the distributed file system (e.g. HDFS). Samza uses an embedded key-value store. With Storm, you’ll have to either roll your own state management at your application layer, or use a higher-level abstraction called Trident.

Use Cases

All three frameworks are particularly well-suited to efficiently process continuous, massive amounts of real-time data. So which one to use? There are no hard rules, at most a few general guidelines.

If you want a high-speed event processing system that allows for incremental computations, Storm would be fine for that. If you further need to run distributed computations on demand, while the client is waiting synchronously for the results, you’ll have Distributed RPC (DRPC) out-of-the-box. Last but not least, because Storm uses Apache Thrift, you can write topologies in any programming language. If you need state persistence and/or exactly-once delivery though, you should look at the higher-level Trident API, which also offers micro-batching.

A few companies using Storm: Twitter, Yahoo!, Spotify, The Weather Channel...

Speaking of micro-batching, if you must have stateful computations, exactly-once delivery and don’t mind a higher latency, you could consider Spark Streaming…specially if you also plan for graph operations, machine learning or SQL access. The Apache Spark stack lets you combine several libraries with streaming (Spark SQL, MLlibGraphX) and provides a convenient unifying programming model. In particular, streaming algorithms (e.g. streaming k-means) allow Spark to facilitate decisions in real-time.


A few companies using Spark: Amazon, Yahoo!, NASA JPL, eBay Inc., Baidu…

If you have a large amount of state to work with (e.g. many gigabytes per partition), Samza co-locates storage and processing on the same machines, allowing to work efficiently with state that won’t fit in memory. The framework also offers flexibility with its pluggable API: its default execution, messaging and storage engines can each be replaced with your choice of alternatives. Moreover, if you have a number of data processing stages from different teams with different codebases, Samza ‘s fine-grained jobs would be particularly well-suited, since they can be added/removed with minimal ripple effects.

A few companies using Samza: LinkedIn, Intuit, Metamarkets, Quantiply, Fortscale…


We only scratched the surface of The Three Apaches. We didn’t cover a number of other features and more subtle differences between these frameworks. Also, it’s important to keep in mind the limits of the above comparisons, as these systems are constantly evolving.

How artificial intelligence is disrupting your organization

26 Feb

robot  women in technology background

Whoever reads a science fiction novel ends up thinking about smart machines that can sense, learn, communicate and interact with human beings. The idea of Artificial Intelligence is not new, but there is a reason if big players like Google, Microsoft or Amazon are betting precisely on this technology right now.
After decades of broken promises, the AI is finally reaching its full potential. It has the power to disrupt your entire business. The question is: How can you harness this technology to shape the future of your organization?

Ever since the human has learned to dream, he has dreamed about ‘automata’, objects able to carry out complex actions automatically. The mythologies of many cultures – Ancient China and Greece, for example – are full of examples of mechanical servants.
Engineers and inventors in different ages attempted to build self-operating machines, resembling animals and humans. Then, in 1920, the Czech writer Karel Čapek used for the first time the term ‘Robot’ to indicate artificial automata.
The rest is history, with the continuing effort to take the final step from mechanical robots to intelligent machines. And here we are, talking about a market expected to reach over five billion dollars by 2020 (Markets & Markets).
The stream of news about the driverless cars, the Internet of Things, and the conversational agents is a clear evidence of the growing interest. Behind the obvious, though, we can find more profitable developments and implications for the Artificial Intelligence.

Back in 2015, while reporting our annual trip at the SXSW, we said that the future of the customer experience goes inevitably through the interconnection of smart objects.
The AI is a top choice when talking about the technologies that will revolutionize the retail store and the physical experience we have with places, products, and people.
The hyperconnected world we live in has a beating heart of chips, wires, and bytes. This is not a science fiction scenario anymore; this is what is happening, here and now, even when you do not see it.
The future of products and services appears more and more linked to the development of intelligent functions and features. Take a look at what has been done already with the embedded AI, that can enable your product to:

  • Communicate with the mobile connected ecosystem – Just think about what we can already do using Google Assistant on the smartphone, or the Amazon Alexa device.
  • Interact with other smart objects that surround us – The Internet of Things has completely changed the way we experience the retail store (and our home, with the domotics).
  • Assist the customer, handling a wider range of requests – The conversational interfaces, like Siri and the chatbots, act as a personal tutor embedded in the device.

As the years pass by, the gap between weak and strong AI widens increasingly. A theory revived by a recent report by Altimeter, not by chance titled “The Age of AI – How Artificial Intelligence Is Transforming Organizations”.
The difference can be defined in terms of the ability to take advantage of the data to learn and improve. Big data and machine learning, in fact, are the two prerequisites of the modern smart technology.
So, on the one hand, we have smart objects that can replace the humans on a specific use case – i.e. to free us from heavy and exhausting duties – but do not learn or evolve in time.
On the other hand, we have the strong AI, the most promising outlook: An intelligence so broad and strong that is able to replicate the general intelligence of human beings. It can mimic the way we think, act and communicate.

The “pure AI” is aspirational but – apart from the Blade Runner charm – this is the field where all the tech giants are willing to bet heavily. The development and implementation of intelligent machines will define the competitive advantage in the age of AI.
According to BCG, “structural flexibility and agility – for both man and machine – become imperative to address the rate and degree of change.


EU Privacy Rules Can Cloud Your IoT Future

24 Feb

When technology companies and communication service providers gather together at the Mobile World Congress (MWC) next week in Barcelona, don’t expect the latest bells-and-whistles of smartphones to stir much industry debate.

Smartphones are maturing.

In contrast, the Internet of Things (IoT) will still be hot. Fueling IoT’s continued momentum is the emergence of fully standardized NB-IoT, a new narrowband radio technology.

However, the market has passed its initial euphoria — when many tech companies and service providers foresaw a brave new world of everything connected to the Internet.

In reality, not everything needs an Internet connection, and not every piece of data – generated by an IoT device – needs a Cloud visit for processing, noted Sami Nassar, vice president of Cybersecurity at NXP Semiconductors, in a recent phone interview with EE Times.

For certain devices such as connected cars, “latency is a killer,” and “security in connectivity is paramount,” he explained. As the IoT market moves to its next phase, “bolting security on top of the Internet type of architecture” won’t be just acceptable, he added.

Looming large for the MWC crowd this year are two unresolved issues: the security and privacy of connected devices, according to Nassar.

GDPR’s Impact on IoT

Whether a connected vehicle, a smart meter or a wearable device, IoT devices are poised to be directly affected by the new General Data Protection Regulation (GDPR), scheduled to take effect in just two years – May 25, 2018.

Companies violating these EU privacy regulations could face penalties of up to 4% of their worldwide revenue (or up to 20 million euros).

In the United States, where many consumers willingly trade their private data for free goods and services, privacy protection might seem an antiquated concept.

Not so in Europe.

There are some basic facts about the GDPR every IoT designer should know.

If you think GDPR is just a European “directive,” you’re mistaken. This is a “regulation” that can take effect without requiring each national government in Europe to pass the enabling legislation.

If you believe GDPR applies to only European companies? Wrong again. The regulation also applies to organizations based outside the EU if they process the personal data of EU residents.

Lastly, if you suspect that GDPR will only affect big data processing companies such as Google, Facebook, Microsoft and Amazon, you’re misled. You aren’t off the hook. Big data processors will be be initially affected first in the “phase one,” said Nassar. Expect “phase two” [of GDPR enforcement] to come down on IoT devices, he added.

EU's GDPR -- a long time in the making (Source: DLA Piper)
Click here for larger image

EU’s GDPR — a long time in the making (Source: DLA Piper)
Click here for larger image

Of course, U.S. consumers are not entirely oblivious to their privacy rights. One reminder was the recent case brought against Vizio. Internet-connected Vizio TV sets were found to be automatically tracking what consumers were watching and transmitting the data to its servers. Consumers didn’t know their TVs were spying on them. When they found out, many objected.

The case against Vizio resulted in a $1.5 million payment to the FTC and an additional civil penalty in New Jersey for a total of $2.2 million.

Although this was seemingly a big victory for consumer rights in the U.S., the penalty could have been a much bigger in Europe. Before the acquisition by LeEco was announced last summer, Vizio had a revenue of $2.9 billion in the year ended in Dec. 2015.

Unlike in the United States where each industry applies and handles violation of privacy rules differently, the EU’s GDPR are sweeping regulations enforced with all industries. A violators like Vizio could have faced much heftier penalty.

What to consider before designing IoT devices
If you design an IoT device, which features and designs must you review and assess to ensure that you are not violating the GDPR?

When we posed the question to DLA Piper, a multinational law firm, its partner Giulio Coraggio told EE Times, “All the aspects of a device that imply the processing of personal data would be relevant.”

Antoon Dierick, lead lawyer at DLA Piper, based in Brussels, added that it’s “important to note that many (if not all) categories of data generated by IoT devices should be considered personal data, given the fact that (a) the device is linked to the user, and (b) is often connected to other personal devices, appliances, apps, etc.” He said, “A good example is a smart electricity meter: the energy data, data concerning the use of the meter, etc. are all considered personal data.”

In particular, as Coraggio noted, the GDPR applies to “the profiling of data, the modalities of usage, the storage period, the security measures implemented, the sharing of data with third parties and others.”

It’s high time now for IoT device designers to “think through” the data their IoT device is collecting and ask if it’s worth that much, said NXP’s Nassar. “Think about privacy by design.”


Why does EU's GDPR matter to IoT technologies? (Source: DLA Piper)

Why does EU’s GDPR matter to IoT technologies? (Source: DLA Piper)

Dierick added that the privacy-by-design principle would “require the manufacturer to market devices which are privacy-friendly by default. This latter aspect will be of high importance for all actors in the IoT value chain.”

Other privacy-by-design principles include: being proactive not reactive, privacy embedded into design, full lifecycle of protection for privacy and security, and being transparent with respect to user privacy (keep it user-centric). After all, the goal of the GDPR is for consumers to control their own data, Nassar concluded.

Unlike big data guys who may find it easy to sign up consumers as long as they offer them what they want in exchange, the story of privacy protection for IoT devices will be different, Nassar cautioned. Consumers are actually paying for an IoT device and the cost of services associated with it. “Enforcement of GDPR will be much tougher on IoT, and consumers will take privacy protection much more seriously,” noted Nassar.

NXP on security, privacy
NXP is positioning itself as a premier chip vendor offering security and privacy solutions for a range of IoT devices.

Many GDPR compliance issues revolve around privacy policies that must be designed into IoT devices and services. To protect privacy, it’s critical for IoT device designers to consider specific implementations related to storage, transfer and processing of data.

NXP’s Nassar explained that one basic principle behind the GDPR is to “disassociate identity from authenticity.” Biometric information in fingerprints, for example, is critical to authenticate the owner of the connected device, but data collected from the device should be processed without linking it to the owner.

Storing secrets — securely
To that end, IoT device designers should ensure that their devices can separately store private or sensitive information — such as biometric templates — from other information left inside the connected device, said Nassar.

At MWC, NXP is rolling out a new embedded Secure Element and NFC solution dubbed PN80T.

PN80T is the first 40nm secure element “to be in mass production and is designed to ease development and implementation of an extended range of secure applications for any platform” including smartphones, wearables to the Internet of Things (IoT), the company explained. Charles Dach, vice president and general manager of mobile transactions at NXP, noted that the PN80T, which is built on the success of NFC applications such as mobile payment and transit, “can be implemented in a range of new security applications that are unrelated to NFC usages.”

In short, NXP is positioning the PN80T as a chip crucial to hardware security for storing secrets.

Key priorities for the framers of the GDPR include secure storage of keys (in tamper resistant HW), individual device identity, secure user identities that respecting a user’s privacy settings, and secure communication channels.

Noting that the PN80T is capable of meeting“security and privacy by design” demands, NXP’s Dach said, “Once you can architect a path to security and isolate it, designing the rest of the platform can move faster.”

Separately, NXP is scheduled to join an MWC panel entitled a “GDPR and the Internet of Things: Protecting the Identity, ‘I’ in the IoT” next week. Others on the panel include representatives from the European Commission, Deutsche Telecom, Qualcomm, an Amsterdam-based law firm called Arthur’s Legal Legal and an advocacy group, Access Now.




What is the difference between Consumer IoT and Industrial IoT (IIoT)?

19 Feb

Internet of Things (IoT) began as an emerging trend and has now become one of the key element of Digital Transformation that is driving the world in many respects.

If your thermostat or refrigerator is connected to the Internet, then it is part of the consumer IoT.  If your factory equipment have sensors connected to internet, then it is part of Industrial IoT(IIoT).

IoT has an impact on end consumers, while IIoT has an impact on industries like Manufacturing, Aviation, Utility, Agriculture, Oil & Gas, Transportation, Energy and Healthcare.

IoT refers to the use of “smart” objects, which are everyday things from cars and home appliances to athletic shoes and light switches that can connect to the Internet, transmitting and receiving data and connecting the physical world to the digital world.

IoT is mostly about human interaction with objects. Devices can alert users when certain events or situations occur or monitor activities:

  • Google Nest sends an alert when temperature in the house dropped below 68 degrees
  • Garage door sensors alert when open
  • Turn up the heat and turn on the driveway lights a half hour before you arrive at your home
  • Meeting room that turns off lights when no one is using it
  • A/C switch off when windows are open

IIoT on the other hand, focus more workers safety, productivity & monitors activities and conditions with remote control functions ability:

  • Drones to monitor oil pipelines
  • Sensors to monitor Chemical factories, drilling equipment, excavators, earth movers
  • Tractors and sprayers in agriculture
  • Smart cities might be a mix of commercial and IIoT.

IoT is important but not critical while IIoT failure often results in life-threatening or other emergency situations.

IIoT provides an unprecedented level of visibility throughout the supply chain. Individual items, cases, pallets, containers and vehicles can be equipped with auto identification tags and tied to GPS-enabled connections to continuously update location and movement.

IoT generates medium or high volume of data while IIoT generates very huge amounts of data (A single turbine compressor blade can generate more than 500GB of data per day) so includes Big Data,Cloud computing, machine learning as necessary computing requirements.

In future, IoT will continue to enhance our lives as consumers while IIoT will enable efficient management of entire supply chain.


Making Sense of Big Data

5 Sep

Table of Contents


  • Arduino – Arduino is an open-source electronics platform based on easy-to-use hardware and software. It’s intended for anyone making interactive projects.
  • BeagleBoard – The BeagleBoard is a low-power open-source hardware single-board computer produced by Texas Instruments in association with Digi-Key and Newark element14.
  • Intel Galileo – The Intel® Galileo Gen 2 board is the first in a family of Arduino*-certified development and prototyping boards based on Intel® architecture and specifically designed for makers, students, educators, and DIY electronics enthusiasts.
  • Microduino – Microduino and mCookie bring powerful, small, stackable electronic hardware to makers, designers, engineers, students and curious tinkerers of all ages. Build open-source projects or create innovative new ones.
  • Node MCU (ESP 8266) – NodeMCU is an open source IoT platform. It uses the Lua scripting language. It is based on the eLua project, and built on the ESP8266 SDK 0.9.5.
  • OLinuXino – OLinuXino is an Open Source Software and Open Source Hardware low cost (EUR 30) Linux Industrial grade single board computer with GPIOs capable of operating from -25°C to +85°C.
  • Particle – A suite of hardware and software tools to help you prototype, scale, and manage your Internet of Things products.
  • Pinoccio – Pinoccio is a pocket-sized, wireless sensor and microcontroller board that combines the features of an Arduino Mega board with a ZigBee compatible 2.4GHz radio.
  • Raspberry Pi – The Raspberry Pi is a low cost, credit-card sized computer that plugs into a computer monitor or TV, and uses a standard keyboard and mouse. It’s capable of doing everything you’d expect a desktop computer to do, from browsing the internet and playing high-definition video, to making spreadsheets, word-processing, and playing games.
  • Tessel – Tessel is a completely open source and community-driven IoT and robotics development platform. It encompases development boards, hardware module add-ons, and the software that runs on them.


Operating systems

  • Apache Mynewt – Apache Mynewt is a real-time, modular operating system for connected IoT devices that need to operate for long periods of time under power, memory, and storage constraints. The first connectivity stack offered is BLE 4.2.
  • ARM mbed – The ARM® mbed™ IoT Device Platform provides the operating system, cloud services, tools and developer ecosystem to make the creation and deployment of commercial, standards-based IoT solutions possible at scale.
  • Contiki – Contiki is an open source operating system for the Internet of Things. Contiki connects tiny low-cost, low-power microcontrollers to the Internet.
  • FreeRTOS – FreeRTOS is a popular real-time operating system kernel for embedded devices, that has been ported to 35 microcontrollers.
  • Google Brillo – Brillo extends the Android platform to all your connected devices, so they are easy to set up and work seamlessly with each other and your smartphone.
  • OpenWrt – OpenWrt is an operating system (in particular, an embedded operating system) based on the Linux kernel, primarily used on embedded devices to route network traffic. The main components are the Linux kernel, util-linux, uClibc or musl, and BusyBox. All components have been optimized for size, to be small enough for fitting into the limited storage and memory available in home routers.
  • Snappy Ubuntu – Snappy Ubuntu Core is a new rendition of Ubuntu with transactional updates. It provides a minimal server image with the same libraries as today’s Ubuntu, but applications are provided through a simpler mechanism.
  • NodeOS – NodeOS is an operating system entirely written in Javascript, and managed by npm on top of the Linux kernel.
  • Raspbian – Raspbian is a free operating system based on Debian optimized for the Raspberry Pi hardware.
  • RIOT – The friendly Operating System for the Internet of Things.
  • Tiny OS – TinyOS is an open source, BSD-licensed operating system designed for low-power wireless devices, such as those used in sensor networks, ubiquitous computing, personal area networks, smart buildings, and smart meters.
  • Windows 10 IoT Core – Windows 10 IoT is a family of Windows 10 editions targeted towards a wide range of intelligent devices, from small industrial gateways to larger more complex devices like point of sales terminals and ATMs.

Programming languages

This sections regroups every awesome programming language, whether it is compiled, interpreted or a DSL, related to embedded development.

  • C – A general-purpose, imperative computer programming language, supporting structured programming, lexical variable scope and recursion, while a static type system prevents many unintended operations.
  • C++ – A general-purpose programming language. It has imperative, object-oriented and generic programming features, while also providing facilities for low-level memory manipulation.
  • Groovy – Groovy is a powerful, optionally typed and dynamic language, with static-typing and static compilation capabilities, for the Java platform aimed at multiplying developers’ productivity thanks to a concise, familiar and easy to learn syntax. It is used by the SmartThings development environment to create smart applications.
  • Lua – Lua is a powerful, fast, lightweight, embeddable scripting language. Lua is dynamically typed, runs by interpreting bytecode for a register-based virtual machine, and has automatic memory management with incremental garbage collection, making it ideal for configuration, scripting, and rapid prototyping.
  • eLua – eLua stands for Embedded Lua and the project offers the full implementation of the Lua Programming Language to the embedded world, extending it with specific features for efficient and portable software embedded development.
  • ELIoT – ELIoT is a very simple and small programming language specifcally designed to facilitate the configuration and control of swarms of small devices such as sensors or actuators.


  • AllJoyn – AllJoyn is an open source software framework that makes it easy for devices and apps to discover and communicate with each other.
  • Apple HomeKit – HomeKit is a framework for communicating with and controlling connected accessories in a user’s home.
  • Countly IoT Analytics – Countly is a general purpose analytics platform for mobile and IoT devices, available as open source.
  • Eclipse Smarthome – The Eclipse SmartHome framework is designed to run on embedded devices, such as a Raspberry Pi, a BeagleBone Black or an Intel Edison. It requires a Java 7 compliant JVM and an OSGi (4.2+) framework, such as Eclipse Equinox.
  • Iotivity – IoTivity is an open source software framework enabling seamless device-to-device connectivity to address the emerging needs of the Internet of Things.
  • Kura – Kura aims at offering a Java/OSGi-based container for M2M applications running in service gateways. Kura provides or, when available, aggregates open source implementations for the most common services needed by M2M applications.
  • Mihini – The main goal of Mihini is to deliver an embedded runtime running on top of Linux, that exposes high-level API for building M2M applications. Mihini aims at enabling easy and portable development, by facilitating access to the I/Os of an M2M system, providing a communication layer, etc.
  • OpenHAB – The openHAB runtime is a set of OSGi bundles deployed on an OSGi framework (Equinox). It is therefore a pure Java solution and needs a JVM to run. Being based on OSGi, it provides a highly modular architecture, which even allows adding and removing functionality during runtime without stopping the service.
  • Gobot – Gobot is a framework for robotics, physical computing, and the Internet of Things, written in the Go programming language.


  • IFTTT – IFTTT is a web-based service that allows users to create chains of simple conditional statements, called “recipes”, which are triggered based on changes to other web services such as Gmail, Facebook, Instagram, and Pinterest. IFTTT is an abbreviation of “If This Then That” (pronounced like “gift” without the “g”).
  • Huginn – Huginn is a system for building agents that perform automated tasks for you online.
  • Kaa – An open-source middleware platform for rapid creation of IoT solutions.

Libraries and Tools

  • Cylon.js – Cylon.js is a JavaScript framework for robotics, physical computing, and the Internet of Things. It makes it incredibly easy to command robots and devices.
  • Luvit – Luvit implements the same APIs as Node.js, but in Lua ! While this framework is not directly involved with IoT development, it is still a great way to rapidly build powertfull, yet memory efficient, embedded web applications.
  • Johnny-Five – Johnny-Five is the original JavaScript Robotics programming framework. Released by Bocoup in 2012, Johnny-Five is maintained by a community of passionate software developers and hardware engineers.
  • WiringPi – WiringPi is a GPIO access library written in C for the BCM2835 used in the Raspberry Pi.
  • Node-RED – A visual tool for wiring the Internet of Things.


  • Amazon Dash – Amazon Dash Button is a Wi-Fi connected device that reorders your favorite item with the press of a button.
  • Freeboard – A real-time interactive dashboard and visualization creator implementing an intuitive drag & drop interface.

Protocols and Networks

Physical layer

 – 802.15.4 (IEEE)

IEEE 802.15.4 is a standard which specifies the physical layer and media access control for low-rate wireless personal area networks (LR-WPANs). It is maintained by the IEEE 802.15 working group, which has defined it in 2003. It is the basis for the ZigBee, ISA100.11a, WirelessHART, and MiWi specifications, each of which further extends the standard by developing the upper layers which are not defined in IEEE 802.15.4. Alternatively, it can be used with 6LoWPAN and standard Internet protocols to build a wireless embedded Internet. – Wikipedia

IEEE standard 802.15.4 intends to offer the fundamental lower network layers of a type of wireless personal area network (WPAN) which focuses on low-cost, low-speed ubiquitous communication between devices. It can be contrasted with other approaches, such as Wi-Fi, which offer more bandwidth and require more power. The emphasis is on very low cost communication of nearby devices with little to no underlying infrastructure, intending to exploit this to lower power consumption even more.

 – Bluetooth (Bluetooth Special Interest Group)

Bluetooth is a wireless technology standard for exchanging data over short distances (using short-wavelength UHF radio waves in the ISM band from 2.4 to 2.485 GHz) from fixed and mobile devices, and building personal area networks (PANs). Invented by telecom vendor Ericsson in 1994, it was originally conceived as a wireless alternative to RS-232 data cables. It can connect several devices, overcoming problems of synchronization. – Wikipedia

Bluetooth is managed by the Bluetooth Special Interest Group (SIG), which has more than 25,000 member companies in the areas of telecommunication, computing, networking, and consumer electronics.

 – Bluetooth Low Energy (Bluetooth Special Interest Group)

Bluetooth low energy (Bluetooth LE, BLE, marketed as Bluetooth Smart) is a wireless personal area network technology designed and marketed by the Bluetooth Special Interest Group aimed at novel applications in the healthcare, fitness, beacons, security, and home entertainment industries. – Wikipedia

Compared to Classic Bluetooth, Bluetooth Smart is intended to provide considerably reduced power consumption and cost while maintaining a similar communication range. The Bluetooth SIG predicts that by 2018 more than 90 percent of Bluetooth-enabled smartphones will support Bluetooth Smart.

 – LoRaWAN (LoRa Alliance)

A LoRaWAN wide area network allows low bit rate communication from and to connected objects, thus participating to Internet of Things, machine-to-machine M2M, and smart city. – Wikipedia

This technology is standardized by the LoRa Alliance. It was initially developed by Cycleo, which was acquired by Semtech in 2012. LoRaWAN is an acronym for Long Range Wide-area network.

 – Sigfox (Sigfox)

Sigfox is a French firm that builds wireless networks to connect low-energy objects such as electricity meters, smart watches, and washing machines, which need to be continuously on and emitting small amounts of data. Its infrastructure is intended to be a contribution to what is known as the Internet of Things (IoT). – Wikipedia

SIGFOX describes itself as “the first and only company providing global cellular connectivity for the Internet of Things.” Its infrastructure is “completely independent of existing networks, such as telecommunications networks.” SIGFOX seeks to provide the means for the “deployment of billions of objects and thousands of new uses” with the long-term goal of “having petabytes of data produced by everyday objects”.

 – Wi-Fi (Wi-Fi Alliance)

Wi-Fi (or WiFi) is a local area wireless computer networking technology that allows electronic devices to network, mainly using the 2.4 gigahertz (12 cm) UHF and 5 gigahertz (6 cm) SHF ISM radio bands. – Wikipedia

The Wi-Fi Alliance defines Wi-Fi as any “wireless local area network” (WLAN) product based on the Institute of Electrical and Electronics Engineers’ (IEEE) 802.11 standards.[1] However, the term “Wi-Fi” is used in general English as a synonym for “WLAN” since most modern WLANs are based on these standards. “Wi-Fi” is a trademark of the Wi-Fi Alliance. The “Wi-Fi Certified” trademark can only be used by Wi-Fi products that successfully complete Wi-Fi Alliance interoperability certification testing.

Network / Transport layer

 – 6LowPan (IETF)

6LoWPAN is an acronym of IPv6 over Low power Wireless Personal Area Networks. 6LoWPAN is the name of a concluded working group in the Internet area of the IETF. – Wikipedia

The 6LoWPAN concept originated from the idea that “the Internet Protocol could and should be applied even to the smallest devices,”and that low-power devices with limited processing capabilities should be able to participate in the Internet of Things. The 6LoWPAN group has defined encapsulation and header compression mechanisms that allow IPv6 packets to be sent and received over IEEE 802.15.4 based networks. IPv4 and IPv6 are the work horses for data delivery for local-area networks, metropolitan area networks, and wide-area networks such as the Internet. Likewise, IEEE 802.15.4 devices provide sensing communication-ability in the wireless domain. The inherent natures of the two networks though, are different.

 – Thread (Thread Group)

Thread is an IPv6 based protocol for “smart” household devices to communicate on a network.

In July 2014 Google Inc’s Nest Labs announced a working group with the companies Samsung, ARM Holdings, Freescale, Silicon Labs, Big Ass Fans and the lock company Yale in an attempt to have Thread become the industry standard by providing Thread certification for products. Other protocols currently in use include ZigBee and Bluetooth Smart. Thread uses 6LoWPAN, which in turn uses the IEEE 802.15.4 wireless protocol with mesh communication, as does ZigBee and other systems. Thread however is IP-addressable, with cloud access and AES encryption. It supports over 250 devices on a network.

 – ZigBee (ZigBee Alliance)

ZigBee is a IEEE 802.15.4-based specification for a suite of high-level communication protocols used to create personal area networks with small, low-power digital radios. – Wikipedia

The technology defined by the ZigBee specification is intended to be simpler and less expensive than other wireless personal area networks (WPANs), such as Bluetooth or Wi-Fi. Applications include wireless light switches, electrical meters with in-home-displays, traffic management systems, and other consumer and industrial equipment that requires short-range low-rate wireless data transfer.

 – Z-Wave (Z-Wave Alliance)

Z-Wave is a wireless communications specification designed to allow devices in the home (lighting, access controls, entertainment systems and household appliances, for example) to communicate with one another for the purposes of home automation. – Wikipedia

Z-Wave technology minimizes power consumption so that it is suitable for battery-operated devices. Z-Wave is designed to provide, reliable, low-latency transmission of small data packets at data rates up to 100kbit/s, unlike Wi-Fi and other IEEE 802.11-based wireless LAN systems that are designed primarily for high data rates. Z-Wave operates in the sub-gigahertz frequency range, around 900 MHz.

Application layer


Constrained Application Protocol (CoAP) is a software protocol intended to be used in very simple electronics devices that allows them to communicate interactively over the Internet. – Wikipedia

CoAP is particularly targeted for small low power sensors, switches, valves and similar components that need to be controlled or supervised remotely, through standard Internet networks. CoAP is an application layer protocol that is intended for use in resource-constrained internet devices, such as WSN nodes.


The Datagram Transport Layer Security (DTLS) communications protocol provides communications security for datagram protocols. – Wikipedia

DTLS allows datagram-based applications to communicate in a way that is designed[by whom?] to prevent eavesdropping, tampering, or message forgery. The DTLS protocol is based on the stream-oriented Transport Layer Security (TLS) protocol and is intended to provide similar security guarantees.

 – Eddystone (Google)

Eddystone is a beacon technology profile released by Google in July 2015. The open source, cross-platform software gives users location and proximity data via Bluetooth low-energy beacon format. – Wikipedia

Though similar to the iBeacon released by Apple in 2013, Eddystone works on both Android and iOS, whereas iBeacon is limited to iOS platforms. A practical application of both softwares is that business owners can target potential customers based on the location of their smartphones in real time.


The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web. – Wikipedia

The standards development of HTTP was coordinated by the Internet Engineering Task Force (IETF) and the World Wide Web Consortium (W3C), culminating in the publication of a series of Requests for Comments (RFCs). The first definition of HTTP/1.1, the version of HTTP in common use, occurred in RFC 2068 in 1997, although this was obsoleted by RFC 2616 in 1999.

 – iBeacon (Apple)

iBeacon is a protocol standardized by Apple and introduced at the Apple Worldwide Developers Conference in 2013. –Wikipedia

iBeacon uses Bluetooth low energy proximity sensing to transmit a universally unique identifier picked up by a compatible app or operating system. The identifier can be used to determine the device’s physical location, track customers, or trigger a location-based action on the device such as a check-in on social media or a push notification.


MQTT (formerly MQ Telemetry Transport) is a publish-subscribe based “light weight” messaging protocol for use on top of the TCP/IP protocol. It is designed for connections with remote locations where a “small code footprint” is required or the network bandwidth is limited. – Wikipedia

The publish-subscribe messaging pattern requires a message broker. The broker is responsible for distributing messages to interested clients based on the topic of a message. Andy Stanford-Clark and Arlen Nipper of Cirrus Link Solutions authored the first version of the protocol in 1999.


Simple (or Streaming) Text Oriented Message Protocol (STOMP), formerly known as TTMP, is a simple text-based protocol, designed for working with message-oriented middleware (MOM). – Wikipedia

STOMP provides an interoperable wire format that allows STOMP clients to talk with any message broker supporting the protocol. It is thus language-agnostic, meaning a broker developed for one programming language or platform can receive communications from client software developed in another language.

 – Websocket

WebSocket is a protocol providing full-duplex communication channels over a single TCP connection. – Wikipedia

WebSocket is designed to be implemented in web browsers and web servers, but it can be used by any client or server application. The WebSocket Protocol is an independent TCP-based protocol. The WebSocket protocol makes more interaction between a browser and a website possible, facilitating live content and the creation of real-time games. This is made possible by providing a standardized way for the server to send content to the browser without being solicited by the client, and allowing for messages to be passed back and forth while keeping the connection open.


Extensible Messaging and Presence Protocol (XMPP) is a communications protocol for message-oriented middleware based on XML (Extensible Markup Language). – Wikipedia

It enables the near-real-time exchange of structured yet extensible data between any two or more network entities. Designed to be extensible, the protocol has also been used for publish-subscribe systems, signalling for VoIP, video, file transfer, gaming, Internet of Things (IoT) applications such as the smart grid, and social networking services.


This sections regroups a curated list of awesome technologies that are closely related to the IoT world.

 – NFC

Near field communication (NFC) is the set of protocols that enable electronic devices to establish radio communication with each other by touching the devices together, or bringing them into proximity to a distance of typically 10cm or less. –Wikipedia


OPC-UA is a not only a protocol for industrial automation but also a technology that allows semantic description and object modeling of industrial environment. Wikipedia

Standards and Alliances


  • ETSI M2M – The ETSI Technical Committee is developing standards for Machine to Machine Communications.
  • OneM2M – The purpose and goal of oneM2M is to develop technical specifications which address the need for a common M2M Service Layer that can be readily embedded within various hardware and software, and relied upon to connect the myriad of devices in the field with M2M application servers worldwide.
  • OPCUA – OPC Unified Architecture (OPC UA) is an industrial M2M communication protocol for interoperability developed by the OPC Foundation.


  • AIOTI – The Internet of Things Innovation (AIOTI) aims to strengthen links and build new relationships between the different IoT players (industries, SMEs, startups) and sectors.
  • AllSeen Alliance – The AllSeen Alliance is a nonprofit consortium dedicated to enabling and driving the widespread adoption of products, systems and services that support the Internet of Everything with an open, universal development framework supported by a vibrant ecosystem and thriving technical community.
  • Bluetooth Special Interest Group – The Bluetooth Special Interest Group (SIG) is the body that oversees the development of Bluetooth standards and the licensing of the Bluetooth technologies and trademarks to manufacturers.
  • IPSO Alliance – The IPSO Alliance provides a foundation for industry growth by fostering awareness, providing education, promoting the industry, generating research, and creating a better understanding of IP and its role in the Internet of Things.
  • LoRa Alliance – The LoRa Alliance is an open, non-profit association of members that believes the internet of things era is now. It was initiated by industry leaders with a mission to standardize Low Power Wide Area Networks (LPWAN) being deployed around the world to enable Internet of Things (IoT), machine-to-machine (M2M), and smart city, and industrial applications.
  • OPC Foundation – The mission of the OPC Foundation is to manage a global organization in which users, vendors and consortia collaborate to create data transfer standards for multi-vendor, multi-platform, secure and reliable interoperability in industrial automation. To support this mission, the OPC Foundation creates and maintains specifications, ensures compliance with OPC specifications via certification testing and collaborates with industry-leading standards organizations.
  • Open Interconnect Consortium – The Open Interconnect Consortium (OIC) is an industry group whose stated mission is to develop standards and certification for devices involved in the Internet of Things (IoT) based around CoAP. OIC was created in July 2014 by Intel, Broadcom, and Samsung Electronics.
  • Thread Group – The Thread Group, composed of members from Nest, Samsung, ARM, Freescale, Silicon Labs, Big Ass Fans and Yale, drives the development of the Thread network protocol.
  • Wi-Fi Alliance – Wi-Fi Alliance® is a worldwide network of companies composed of several companies forming a global non-profit association with the goal of driving the best user experience with a new wireless networking technology – regardless of brand.
  • Zigbee Alliance – The ZigBee Alliance is an open, non-profit association of approximately 450 members driving development of innovative, reliable and easy-to-use ZigBee standards.
  • Z-Wave Alliance – Established in 2005, the Z-Wave Alliance is comprised of industry leaders throughout the globe that are dedicated to the development and extension of Z-Wave as the key enabling technology for ‘smart’ home and business applications.



Abusing the Internet of Things: Blackouts, Freakouts, and Stakeouts (2015) by Nitesh Dhanjani [5.0]

future with billions of connected “things” includes monumental security concerns. This practical book explores how malicious attackers can abuse popular IoT-based devices, including wireless LED lightbulbs, electronic door locks, baby monitors, smart TVs, and connected cars.

Building Wireless Sensor Networks: with ZigBee, XBee, Arduino, and Processing (2011) by Robert Faludi [4.5]

Get ready to create distributed sensor systems and intelligent interactive devices using the ZigBee wireless networking protocol and Series 2 XBee radios. By the time you’re halfway through this fast-paced, hands-on guide, you’ll have built a series of useful projects, including a complete ZigBee wireless network that delivers remotely sensed data.

Designing the Internet of Things (2013) by Adrian McEwen and Hakim Cassimally [4.0]

Whether it’s called physical computing, ubiquitous computing, or the Internet of Things, it’s a hot topic in technology: how to channel your inner Steve Jobs and successfully combine hardware, embedded software, web services, electronics, and cool design to create cutting-edge devices that are fun, interactive, and practical. If you’d like to create the next must-have product, this unique book is the perfect place to start.

Getting Started with Bluetooth Low Energy: Tools and Techniques for Low-Power Networking (2014) by Kevin Townsend,Carles CufíAkiba and Robert Davidson [4.5]

This book provides a solid, high-level overview of how devices use Ble to communicate with each other. You’ll learn useful low-cost tools for developing and testing Ble-enabled mobile apps and embedded firmware and get examples using various development platforms including iOs and Android for app developers and embedded platforms for product designers and hardware engineers.

Smart Things: Ubiquitous Computing User Experience Design (2010) by Mike Kuniavsky [4.5]

Smart Things presents a problem-solving approach to addressing designers’ needs and concentrates on process, rather than technological detail, to keep from being quickly outdated. It pays close attention to the capabilities and limitations of the medium in question and discusses the tradeoffs and challenges of design in a commercial environment.




Big Data, IoT & Blockchain: Ready to Follow the Yellow Brick Road?

24 Mar

Disruption. You can’t have a discussion today about business or technology without the term entering the conversation. I think it’s become an unwritten rule. It’s almost as if no one will take you seriously unless you’re talking about business disruption. Or how disruptive technologies can be used to advance business and provide a competitive edge.

Take Big Data and the Internet of Things (IoT). Both rank highly on the list of disruptive technologies. And as with most technologies, there are areas of great synergy that ultimately provide a yellow brick road to real business value. (See my recent blog Big Data, the Internet of Things, and Russian Nesting Dolls.)

Gold ingod road in grass with sky

Blockchain enters the disruptive dialogue

But recently, a new topic has enlivened the disruption discussions: Blockchain technology. And with it, the requisite stream of questions. What exactly is it? How does it help (or does it help) provide business value? How will it affect my current initiatives? And are there synergies to be had—or do I have to worry about it blowing everything up?

What is blockchain—and how is it associated with Bitcoin?

If you do a Google search on blockchain, you’ll find several results that inevitably pair the terms “blockchain” and “Bitcoin.” That’s because blockchain technology enables digital currencies like Bitcoin to work. As you may be aware, Bitcoin has no physical form, is not controlled by a single entity, nor is it backed by any government or agency.

(I’m not going to attempt to discuss the pros and cons of Bitcoin here. Those conversations can be almost as emotional as political discussions—and voluminous enough to fill books.)

A permanent digital transaction database…

In simple terms, blockchain is a digital ledger of transactions that you might think of as a spreadsheet. Yet it comprises a constantly growing list of transactions called “blocks”—all of which are sequentially connected. Each block has a link to the previous one in the list. Once a block is in the chain it can’t be removed, so it becomes part of a permanent database containing all the transactions that have occurred since its inception.

…is also the ultimate distributed database

But perhaps the most interesting thing about blockchain is that there’s no central authority or single source of the database. Which means it exists on every system that’s associated with it. Yes, every system has its own complete copy of the blockchain. As new blocks are added, they’re also received by every system—for the ultimate distributed database. So if you lose your copy, no problem. By rejoining the blockchain network you get a fresh new copy of the entire blockchain.

But how do you ensure transactional security?

By now you’re probably wondering, “How can this possibly result in a secure method for conducting digital transactions?” The short answer is through some very complex cryptography, math puzzles, and crowdsourcing consensus. There’s a great video that explains it in some detail on YouTube. It’s a little over 20 minutes long, but is the best explanation I’ve seen of a very complex solution.

The net result is called a “trustless system.” Which is not to say the system can’t be trusted. It simply means that two parties don’t need a trusted third party (such as a bank or credit card company) to maintain a ledger and valid transactions. Because every transaction can always be verified against the distributed ledger, a copy of which resides with all parties.

Note: One thing that’s important to understand is that while you can’t have Bitcoin without blockchain, you can use blockchain without involving Bitcoin—and that’s when things can become very interesting.

Blockchain and Big Data

When you talk about blockchain in the context of Bitcoin, the connection to Big Data seems a little tenuous. What if, instead of Bitcoin, the blockchain was a ledger for other financial transactions? Or business contracts? Or stock trades?

The financial services industry is starting to take a serious look at block chain technology. Citi, Nasdaq, and Visa recently made significant , a Bitcoin blockchain service provider. And Oliver Bussmann, CIO of UBS says that blockchain technology could “pare transaction processing time from days to minutes.”

The business imperative in financial services for blockchain is powerful. Imagine blockchains of that magnitude. Huge data lakes of blocks that contain the full history of every financial transaction, all available for analysis. Blockchain provides for the integrity of the ledger, but not for the analysis. That’s where Big Data and accompanying analysis tools will come into play.

Blockchain and the Internet of Things

There’s no doubt that IoT is a tremendous growth industry. Gartner predicts that the number of “things” will exceed 25 billion (with a B) devices within the next four years. These things can be anything from a small sensor to a large appliance—and everything in between. Two key challenges are securing those devices, and the privacy of the data they exchange.

Traditional centralized authority and message brokering could help address these issues, but will not scale with the number of devices predicted and the 100’s of billions of transactions the devices will generate.

Several major industry leaders put forth blockchain technology is as a possible solution to these challenges. The vision is a decentralized IoT, where the blockchain can act as the framework for facilitating transaction processing and coordination among interacting devices. Each device would manage its own roles and behavior and rules for interaction.

Follow the Yellow Brick Road

The blockchain builds itself a block at a time, always growing and moving forward, but also maintaining the trail of where it’s been. While the blockchain’s original purpose was in support of Bitcoin digital currency, like most disruptive technologies its value is growing in unexpected ways and directions.

As a technologist, I find the technology fascinating. That being said, technology is just a tool. It’s our responsibility to ensure the tools can be leveraged to provide true business value. Whethers its reduction of transaction processing time, analysis of transaction trends, or providing a mechanism to securely scale the Internet of Things messaging, the synergies with Big Data and IoT are one way we can follow that yellow block road to true business value.

This post is sponsored by SAS and Big Data Forum.


%d bloggers like this: